{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "13_Recurrent_Neural_Networks",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "metadata": {
        "id": "bOChJSNXtC9g",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Recurrent Neural Networks"
      ]
    },
    {
      "metadata": {
        "id": "OLIxEDq6VhvZ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "<img src=\"https://raw.githubusercontent.com/GokuMohandas/practicalAI/master/images/logo.png\" width=150>\n",
        "\n",
        "When working with sequential data (time-series, sentences, etc.) the order of the inputs is crucial for the task at hand. Recurrent neural networks (RNNs) process sequential data by accounting for the current input and also what has been learned from previous inputs. In this notebook, we'll learn how to create and train RNNs on sequential data.\n",
        "\n",
        "<img src=\"https://raw.githubusercontent.com/GokuMohandas/practicalAI/master/images/rnn.png\" width=550>\n",
        "\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "VoMq0eFRvugb",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Overview"
      ]
    },
    {
      "metadata": {
        "id": "qWro5T5qTJJL",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "* **Objective:**  Process sequential data by accounting for the currend input and also what has been learned from previous inputs.\n",
        "* **Advantages:** \n",
        "    * Account for order and previous inputs in a meaningful way.\n",
        "    * Conditioned generation for generating sequences.\n",
        "* **Disadvantages:** \n",
        "    * Each time step's prediction depends on the previous prediction so it's difficult to parallelize RNN operations. \n",
        "    * Processing long sequences can yield memory and computation issues.\n",
        "    * Interpretability is difficult but there are few [techniques](https://arxiv.org/abs/1506.02078) that use the activations from RNNs to see what parts of the inputs are processed. \n",
        "* **Miscellaneous:** \n",
        "    * Architectural tweaks to make RNNs faster and interpretable is an ongoing area of research."
      ]
    },
    {
      "metadata": {
        "id": "rsHeBbehrKzl",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "<img src=\"https://raw.githubusercontent.com/GokuMohandas/practicalAI/master/images/rnn2.png\" width=650>\n",
        "\n",
        "RNN forward pass for a single time step $X_t$:\n",
        "\n",
        "$h_t = tanh(W_{hh}h_{t-1} + W_{xh}X_t+b_h)$\n",
        "\n",
        "$y_t = W_{hy}h_t + b_y $\n",
        "\n",
        "$ P(y) = softmax(y_t) = \\frac{e^y}{\\sum e^y} $\n",
        "\n",
        "*where*:\n",
        "* $X_t$ = input at time step t | $\\in \\mathbb{R}^{NXE}$ ($N$ is the batch size, $E$ is the embedding dim)\n",
        "* $W_{hh}$ = hidden units weights| $\\in \\mathbb{R}^{HXH}$ ($H$ is the hidden dim)\n",
        "* $h_{t-1}$ = previous timestep's hidden state $\\in \\mathbb{R}^{NXH}$\n",
        "* $W_{xh}$ = input weights| $\\in \\mathbb{R}^{EXH}$\n",
        "* $b_h$ = hidden units bias $\\in \\mathbb{R}^{HX1}$\n",
        "* $W_{hy}$ = output weights| $\\in \\mathbb{R}^{HXC}$ ($C$ is the number of classes)\n",
        "* $b_y$ = output bias $\\in \\mathbb{R}^{CX1}$\n",
        "\n",
        "You repeat this for every time step's input ($X_{t+1}, X_{t+2}, ..., X_{N})$ to the get the predicted outputs at each time step.\n",
        "\n",
        "**Note**: At the first time step, the previous hidden state $h_{t-1}$ can either be a zero vector (unconditioned) or initialize (conditioned). If we are conditioning the RNN, the first hidden state $h_0$ can belong to a specific condition or we can concat the specific condition to the randomly initialized hidden vectors at each time step. More on this in the subsequent notebooks on RNNs."
      ]
    },
    {
      "metadata": {
        "id": "dIXlGMExJD6w",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Let's see what the forward pass looks like with an RNN for a synthetic task such as processing reviews (a sequence of words) to predict the sentiment at the end of processing the review."
      ]
    },
    {
      "metadata": {
        "id": "RcWE5cw0_cKA",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Load PyTorch library\n",
        "!pip3 install torch"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "o6eEK1wM_dXG",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import torch\n",
        "import torch.nn as nn\n",
        "import torch.nn.functional as F"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Qi9hIEV6COLF",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "batch_size = 5\n",
        "seq_size = 10 # max length per input (masking will be used for sequences that aren't this max length)\n",
        "x_lengths = [8, 5, 4, 10, 5] # lengths of each input sequence\n",
        "embedding_dim = 100\n",
        "rnn_hidden_dim = 256\n",
        "output_dim = 4"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "bLEzfxjhB94C",
        "colab_type": "code",
        "outputId": "a2952225-5c5f-4250-ec79-6288dbf34c42",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "cell_type": "code",
      "source": [
        "# Initialize synthetic inputs\n",
        "x_in = torch.randn(batch_size, seq_size, embedding_dim)\n",
        "x_lengths = torch.tensor(x_lengths)\n",
        "print (x_in.size())"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "torch.Size([5, 10, 100])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "dr6oLqtXB98N",
        "colab_type": "code",
        "outputId": "b54ff9e2-2857-499c-b739-c2c15e869977",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "cell_type": "code",
      "source": [
        "# Initialize hidden state\n",
        "hidden_t = torch.zeros((batch_size, rnn_hidden_dim))\n",
        "print (hidden_t.size())"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "torch.Size([5, 256])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "ryZMOLLgB9-v",
        "colab_type": "code",
        "outputId": "1bf649dd-4c47-4686-f15e-a9b408e32e61",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "cell_type": "code",
      "source": [
        "# Initialize RNN cell\n",
        "rnn_cell = nn.RNNCell(embedding_dim, rnn_hidden_dim)\n",
        "print (rnn_cell)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "RNNCell(100, 256)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "rlbZ7ujxExXb",
        "colab_type": "code",
        "outputId": "9d6be9d0-20bc-4be8-c155-8d098ce5cbe1",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "cell_type": "code",
      "source": [
        "# Forward pass through RNN\n",
        "x_in = x_in.permute(1, 0, 2) # RNN needs batch_size to be at dim 1\n",
        "\n",
        "# Loop through the inputs time steps\n",
        "hiddens = []\n",
        "for t in range(seq_size):\n",
        "    hidden_t = rnn_cell(x_in[t], hidden_t)\n",
        "    hiddens.append(hidden_t)\n",
        "hiddens = torch.stack(hiddens)\n",
        "hiddens = hiddens.permute(1, 0, 2) # bring batch_size back to dim 0\n",
        "print (hiddens.size())"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "torch.Size([5, 10, 256])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "3TTL-jmg-MHa",
        "colab_type": "code",
        "outputId": "22524f37-2dd4-472e-b76d-404eed0250a7",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "cell_type": "code",
      "source": [
        "# We also could've used a more abstracted layer\n",
        "x_in = torch.randn(batch_size, seq_size, embedding_dim)\n",
        "rnn = nn.RNN(embedding_dim, rnn_hidden_dim, batch_first=True)\n",
        "out, h_n = rnn(x_in) #h_n is the last hidden state\n",
        "print (\"out: \", out.size())\n",
        "print (\"h_n: \", h_n.size())"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "out:  torch.Size([5, 10, 256])\n",
            "h_n:  torch.Size([1, 5, 256])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "iAsyRNnbHwcT",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def gather_last_relevant_hidden(hiddens, x_lengths):\n",
        "    x_lengths = x_lengths.long().detach().cpu().numpy() - 1\n",
        "    out = []\n",
        "    for batch_index, column_index in enumerate(x_lengths):\n",
        "        out.append(hiddens[batch_index, column_index])\n",
        "    return torch.stack(out)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "PVhp1KLqHqpA",
        "colab_type": "code",
        "outputId": "f4b0708b-88dc-4b28-93c0-073bbcb6fe2a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "cell_type": "code",
      "source": [
        "# Gather the last relevant hidden state\n",
        "z = gather_last_relevant_hidden(hiddens, x_lengths)\n",
        "print (z.size())"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "torch.Size([5, 256])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "yGk_iZ5cITZl",
        "colab_type": "code",
        "outputId": "082e0035-4b31-4066-f02a-6686e2070bdd",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 119
        }
      },
      "cell_type": "code",
      "source": [
        "# Forward pass through FC layer\n",
        "fc1 = nn.Linear(rnn_hidden_dim, output_dim)\n",
        "y_pred = fc1(z)\n",
        "y_pred = F.softmax(y_pred, dim=1)\n",
        "print (y_pred.size())\n",
        "print (y_pred)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "torch.Size([5, 4])\n",
            "tensor([[0.2963, 0.2147, 0.2409, 0.2482],\n",
            "        [0.2638, 0.3373, 0.1835, 0.2154],\n",
            "        [0.2820, 0.2741, 0.1941, 0.2498],\n",
            "        [0.1895, 0.3276, 0.2353, 0.2476],\n",
            "        [0.2600, 0.2098, 0.2310, 0.2992]], grad_fn=<SoftmaxBackward>)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "hPBQpki_n6yY",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Sequential data"
      ]
    },
    {
      "metadata": {
        "id": "kP1awuluoCSr",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "There are a variety of different sequential tasks that RNNs can help with.\n",
        "\n",
        "1. **One to one**: there is one input and produces one output. \n",
        "    * Ex. Given a word predict it's class (verb, noun, etc.).\n",
        "2. **One to many**: one input generates many outputs.\n",
        "    * Ex. Given a sentiment (positive, negative, etc.) generate a review.\n",
        "3. **Many to one**: Many inputs are sequentially processed to generate one output.\n",
        "    * Ex. Process the words in a review to predict the sentiment.\n",
        "4. **Many to many**: Many inputs are sequentially processed to generate many outputs.\n",
        "    * Ex. Given a sentence in French, processes the entire sentence and then generate the English translation.\n",
        "    * Ex. Given a sequence of time-series data, predict the probability of an event (risk of disease) at each time step.\n",
        "\n",
        "<img src=\"https://raw.githubusercontent.com/GokuMohandas/practicalAI/master/images/seq2seq.jpeg\" width=700>"
      ]
    },
    {
      "metadata": {
        "id": "tnxUIEMdukYY",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Issues with vanilla RNNs"
      ]
    },
    {
      "metadata": {
        "id": "uMx2s93VLUTt",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "There are several issues with the vanilla RNN that we've seen so far. \n",
        "\n",
        "1. When we have an input sequence that has many time steps, it becomes difficult for the model to retain information seen earlier as we process more and more of the downstream timesteps. The goals of the model is to retain the useful components in the previously seen time steps but this becomes cumbersome when we have so many time steps to process. \n",
        "\n",
        "2. During backpropagation, the gradient from the loss has to travel all the way back towards the first time step. If our gradient is larger than 1 (${1.01}^{1000} = 20959$) or less than 1 (${0.99}^{1000} = 4.31e-5$) and we have lot's of time steps, this can quickly spiral out of control.\n",
        "\n",
        "To address both these issues, the concept of gating was introduced to RNNs. Gating allows RNNs to control the information flow between each time step to optimize on the task. Selectively allowing information to pass through allows the model to process inputs with many time steps. The most common RNN gated varients are the long short term memory ([LSTM](https://pytorch.org/docs/stable/nn.html#torch.nn.LSTM)) units and gated recurrent units ([GRUs](https://pytorch.org/docs/stable/nn.html#torch.nn.GRU)). You can read more about how these units work [here](http://colah.github.io/posts/2015-08-Understanding-LSTMs/).\n",
        "\n",
        "<img src=\"https://raw.githubusercontent.com/GokuMohandas/practicalAI/master/images/gates.png\" width=900>"
      ]
    },
    {
      "metadata": {
        "id": "tirko0kwp-9J",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# GRU in PyTorch\n",
        "gru = nn.GRU(input_size=embedding_dim, hidden_size=rnn_hidden_dim, \n",
        "             batch_first=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "UZjUhh4VBWxM",
        "colab_type": "code",
        "outputId": "07a0ab92-2e80-458d-f81d-782450b9094b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "cell_type": "code",
      "source": [
        "# Initialize synthetic input\n",
        "x_in = torch.randn(batch_size, seq_size, embedding_dim)\n",
        "print (x_in.size())"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "torch.Size([5, 10, 100])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "xJ_SE7AvBfa4",
        "colab_type": "code",
        "outputId": "fb4cb2b3-72a2-4609-985c-0b23a1decdbb",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "cell_type": "code",
      "source": [
        "# Forward pass\n",
        "out, h_n = gru(x_in)\n",
        "print (\"out:\", out.size())\n",
        "print (\"h_n:\", h_n.size())"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "out: torch.Size([5, 10, 256])\n",
            "h_n: torch.Size([1, 5, 256])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "ij_GA2Rr9BbA",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "**Note**: Choosing whether to use GRU or LSTM really depends on the data and empirical performance. GRUs offer comparable performance with reduce number of parameters while LSTMs are more efficient and may make the difference in performance for your particular task."
      ]
    },
    {
      "metadata": {
        "id": "9agJw4gwK1LC",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Bidirectional RNNs"
      ]
    },
    {
      "metadata": {
        "id": "Xck0n-KpmXkV",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "There have been many advancements with RNNs ([attention](https://www.oreilly.com/ideas/interpretability-via-attentional-and-memory-based-interfaces-using-tensorflow), Quasi RNNs, etc.) that we will cover in later lessons but one of the basic and widely used ones are bidirectional RNNs (Bi-RNNs). The motivation behind bidirectional RNNs is to process an input sequence by both directions. Accounting for context from both sides can aid in performance when the entire input sequence is known at time of inference. A common application of Bi-RNNs is in translation where it's advantageous to look at an entire sentence from both sides when translating to another language (ie. Japanese → English).\n",
        "\n",
        "<img src=\"https://raw.githubusercontent.com/GokuMohandas/practicalAI/master/images/birnn.png\" width=700>"
      ]
    },
    {
      "metadata": {
        "id": "gSk_5XrvApCd",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# BiGRU in PyTorch\n",
        "bi_gru = nn.GRU(input_size=embedding_dim, hidden_size=rnn_hidden_dim, \n",
        "                batch_first=True, bidirectional=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Fx7-GTptBCtZ",
        "colab_type": "code",
        "outputId": "6150454c-e67c-4b9f-e60e-6b963916ce05",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "cell_type": "code",
      "source": [
        "# Forward pass\n",
        "out, h_n = bi_gru(x_in)\n",
        "print (\"out:\", out.size()) # collection of all hidden states from the RNN for each time step\n",
        "print (\"h_n:\", h_n.size()) # last hidden state from the RNN"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "out: torch.Size([5, 10, 512])\n",
            "h_n: torch.Size([2, 5, 256])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "k5lvJirLBjI6",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Notice that the output for each sample at each timestamp has size 512 (double the hidden dim). This is because this includes both the forward and backward directions from the BiRNN. "
      ]
    },
    {
      "metadata": {
        "id": "mJSknbofK2S9",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Document classification with RNNs"
      ]
    },
    {
      "metadata": {
        "id": "JgYdEZmHlmft",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Let's apply RNNs to the document classification task from the [emebddings notebook](https://colab.research.google.com/drive/1yDa5ZTqKVoLl-qRgH-N9xs3pdrDJ0Fb4) where we want to predict an article's category given its title."
      ]
    },
    {
      "metadata": {
        "id": "eIvXqvPQEiDC",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Set up"
      ]
    },
    {
      "metadata": {
        "id": "muTcvMynlmAu",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import os\n",
        "from argparse import Namespace\n",
        "import collections\n",
        "import copy\n",
        "import json\n",
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "import re\n",
        "import torch"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "00ESjecep-_y",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Set Numpy and PyTorch seeds\n",
        "def set_seeds(seed, cuda):\n",
        "    np.random.seed(seed)\n",
        "    torch.manual_seed(seed)\n",
        "    if cuda:\n",
        "        torch.cuda.manual_seed_all(seed)\n",
        "        \n",
        "# Creating directories\n",
        "def create_dirs(dirpath):\n",
        "    if not os.path.exists(dirpath):\n",
        "        os.makedirs(dirpath)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "m67THDvxEl1e",
        "colab_type": "code",
        "outputId": "414a482e-817b-4940-92ec-229596343b7e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "cell_type": "code",
      "source": [
        "# Arguments\n",
        "args = Namespace(\n",
        "    seed=1234,\n",
        "    cuda=True,\n",
        "    shuffle=True,\n",
        "    data_file=\"news.csv\",\n",
        "    vectorizer_file=\"vectorizer.json\",\n",
        "    model_state_file=\"model.pth\",\n",
        "    save_dir=\"news\",\n",
        "    train_size=0.7,\n",
        "    val_size=0.15,\n",
        "    test_size=0.15,\n",
        "    pretrained_embeddings=None,\n",
        "    cutoff=25, # token must appear at least <cutoff> times to be in SequenceVocabulary\n",
        "    num_epochs=5,\n",
        "    early_stopping_criteria=5,\n",
        "    learning_rate=1e-3,\n",
        "    batch_size=64,\n",
        "    embedding_dim=100,\n",
        "    rnn_hidden_dim=128,\n",
        "    hidden_dim=100,\n",
        "    num_layers=1,\n",
        "    bidirectional=False,\n",
        "    dropout_p=0.1,\n",
        ")\n",
        "\n",
        "# Set seeds\n",
        "set_seeds(seed=args.seed, cuda=args.cuda)\n",
        "\n",
        "# Create save dir\n",
        "create_dirs(args.save_dir)\n",
        "\n",
        "# Expand filepaths\n",
        "args.vectorizer_file = os.path.join(args.save_dir, args.vectorizer_file)\n",
        "args.model_state_file = os.path.join(args.save_dir, args.model_state_file)\n",
        "\n",
        "# Check CUDA\n",
        "if not torch.cuda.is_available():\n",
        "    args.cuda = False\n",
        "args.device = torch.device(\"cuda\" if args.cuda else \"cpu\")\n",
        "print(\"Using CUDA: {}\".format(args.cuda))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Using CUDA: True\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "s7T-_kGvExVW",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Data"
      ]
    },
    {
      "metadata": {
        "id": "XVyK25xOEwjN",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import re\n",
        "import urllib"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "M_gclwECEwll",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Upload data from GitHub to notebook's local drive\n",
        "url = \"https://raw.githubusercontent.com/GokuMohandas/practicalAI/master/data/news.csv\"\n",
        "response = urllib.request.urlopen(url)\n",
        "html = response.read()\n",
        "with open(args.data_file, 'wb') as fp:\n",
        "    fp.write(html)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "V244zOIPEwoP",
        "colab_type": "code",
        "outputId": "6c7f9959-362b-43db-80d9-807f9595c4c1",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        }
      },
      "cell_type": "code",
      "source": [
        "# Raw data\n",
        "df = pd.read_csv(args.data_file, header=0)\n",
        "df.head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>category</th>\n",
              "      <th>title</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Business</td>\n",
              "      <td>Wall St. Bears Claw Back Into the Black (Reuters)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Business</td>\n",
              "      <td>Carlyle Looks Toward Commercial Aerospace (Reu...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Business</td>\n",
              "      <td>Oil and Economy Cloud Stocks' Outlook (Reuters)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Business</td>\n",
              "      <td>Iraq Halts Oil Exports from Main Southern Pipe...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Business</td>\n",
              "      <td>Oil prices soar to all-time record, posing new...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   category                                              title\n",
              "0  Business  Wall St. Bears Claw Back Into the Black (Reuters)\n",
              "1  Business  Carlyle Looks Toward Commercial Aerospace (Reu...\n",
              "2  Business    Oil and Economy Cloud Stocks' Outlook (Reuters)\n",
              "3  Business  Iraq Halts Oil Exports from Main Southern Pipe...\n",
              "4  Business  Oil prices soar to all-time record, posing new..."
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 22
        }
      ]
    },
    {
      "metadata": {
        "id": "ICl2MNK4EwrL",
        "colab_type": "code",
        "outputId": "0669edde-c740-457a-e342-70d10e485c72",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 85
        }
      },
      "cell_type": "code",
      "source": [
        "# Split by category\n",
        "by_category = collections.defaultdict(list)\n",
        "for _, row in df.iterrows():\n",
        "    by_category[row.category].append(row.to_dict())\n",
        "for category in by_category:\n",
        "    print (\"{0}: {1}\".format(category, len(by_category[category])))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Business: 30000\n",
            "Sci/Tech: 30000\n",
            "Sports: 30000\n",
            "World: 30000\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "76PwKQHLEww5",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Create split data\n",
        "final_list = []\n",
        "for _, item_list in sorted(by_category.items()):\n",
        "    if args.shuffle:\n",
        "        np.random.shuffle(item_list)\n",
        "    n = len(item_list)\n",
        "    n_train = int(args.train_size*n)\n",
        "    n_val = int(args.val_size*n)\n",
        "    n_test = int(args.test_size*n)\n",
        "\n",
        "  # Give data point a split attribute\n",
        "    for item in item_list[:n_train]:\n",
        "        item['split'] = 'train'\n",
        "    for item in item_list[n_train:n_train+n_val]:\n",
        "        item['split'] = 'val'\n",
        "    for item in item_list[n_train+n_val:]:\n",
        "        item['split'] = 'test'  \n",
        "\n",
        "    # Add to final list\n",
        "    final_list.extend(item_list)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "CQeS0KHOEwzm",
        "colab_type": "code",
        "outputId": "a2a8ed57-da3c-4441-9159-08e3d66e1a80",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 85
        }
      },
      "cell_type": "code",
      "source": [
        "# df with split datasets\n",
        "split_df = pd.DataFrame(final_list)\n",
        "split_df[\"split\"].value_counts()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "train    84000\n",
              "val      18000\n",
              "test     18000\n",
              "Name: split, dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 25
        }
      ]
    },
    {
      "metadata": {
        "id": "pPJDyVusEw3-",
        "colab_type": "code",
        "outputId": "63051d89-3ea7-44d1-cc5a-5655e904507f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        }
      },
      "cell_type": "code",
      "source": [
        "# Preprocessing\n",
        "def preprocess_text(text):\n",
        "    text = ' '.join(word.lower() for word in text.split(\" \"))\n",
        "    text = re.sub(r\"([.,!?])\", r\" \\1 \", text)\n",
        "    text = re.sub(r\"[^a-zA-Z.,!?]+\", r\" \", text)\n",
        "    text = text.strip()\n",
        "    return text\n",
        "    \n",
        "split_df.title = split_df.title.apply(preprocess_text)\n",
        "split_df.head()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>category</th>\n",
              "      <th>split</th>\n",
              "      <th>title</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Business</td>\n",
              "      <td>train</td>\n",
              "      <td>general electric posts higher rd quarter profit</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Business</td>\n",
              "      <td>train</td>\n",
              "      <td>lilly to eliminate up to us jobs</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Business</td>\n",
              "      <td>train</td>\n",
              "      <td>s amp p lowers america west outlook to negative</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Business</td>\n",
              "      <td>train</td>\n",
              "      <td>does rand walk the talk on labor policy ?</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Business</td>\n",
              "      <td>train</td>\n",
              "      <td>housekeeper advocates for changes</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   category  split                                            title\n",
              "0  Business  train  general electric posts higher rd quarter profit\n",
              "1  Business  train                 lilly to eliminate up to us jobs\n",
              "2  Business  train  s amp p lowers america west outlook to negative\n",
              "3  Business  train        does rand walk the talk on labor policy ?\n",
              "4  Business  train                housekeeper advocates for changes"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 26
        }
      ]
    },
    {
      "metadata": {
        "id": "NHzGXAI3E7lF",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Vocabulary"
      ]
    },
    {
      "metadata": {
        "id": "ZIRUjX0MEw88",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "class Vocabulary(object):\n",
        "    def __init__(self, token_to_idx=None):\n",
        "\n",
        "        # Token to index\n",
        "        if token_to_idx is None:\n",
        "            token_to_idx = {}\n",
        "        self.token_to_idx = token_to_idx\n",
        "\n",
        "        # Index to token\n",
        "        self.idx_to_token = {idx: token \\\n",
        "                             for token, idx in self.token_to_idx.items()}\n",
        "\n",
        "    def to_serializable(self):\n",
        "        return {'token_to_idx': self.token_to_idx}\n",
        "\n",
        "    @classmethod\n",
        "    def from_serializable(cls, contents):\n",
        "        return cls(**contents)\n",
        "\n",
        "    def add_token(self, token):\n",
        "        if token in self.token_to_idx:\n",
        "            index = self.token_to_idx[token]\n",
        "        else:\n",
        "            index = len(self.token_to_idx)\n",
        "            self.token_to_idx[token] = index\n",
        "            self.idx_to_token[index] = token\n",
        "        return index\n",
        "\n",
        "    def add_tokens(self, tokens):\n",
        "        return [self.add_token[token] for token in tokens]\n",
        "\n",
        "    def lookup_token(self, token):\n",
        "        return self.token_to_idx[token]\n",
        "\n",
        "    def lookup_index(self, index):\n",
        "        if index not in self.idx_to_token:\n",
        "            raise KeyError(\"the index (%d) is not in the Vocabulary\" % index)\n",
        "        return self.idx_to_token[index]\n",
        "\n",
        "    def __str__(self):\n",
        "        return \"<Vocabulary(size=%d)>\" % len(self)\n",
        "\n",
        "    def __len__(self):\n",
        "        return len(self.token_to_idx)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "1LtYf3lpExBb",
        "colab_type": "code",
        "outputId": "1d2f3947-6662-4dec-8df7-724967589d3a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 85
        }
      },
      "cell_type": "code",
      "source": [
        "# Vocabulary instance\n",
        "category_vocab = Vocabulary()\n",
        "for index, row in df.iterrows():\n",
        "    category_vocab.add_token(row.category)\n",
        "print (category_vocab) # __str__\n",
        "print (len(category_vocab)) # __len__\n",
        "index = category_vocab.lookup_token(\"Business\")\n",
        "print (index)\n",
        "print (category_vocab.lookup_index(index))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<Vocabulary(size=4)>\n",
            "4\n",
            "0\n",
            "Business\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "Z0zkF6CsE_yH",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Sequence vocabulary"
      ]
    },
    {
      "metadata": {
        "id": "QtntaISyE_1c",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Next, we're going to create our Vocabulary classes for the article's title, which is a sequence of tokens."
      ]
    },
    {
      "metadata": {
        "id": "ovI8QRefEw_p",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from collections import Counter\n",
        "import string"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "4W3ZouuTEw1_",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "class SequenceVocabulary(Vocabulary):\n",
        "    def __init__(self, token_to_idx=None, unk_token=\"<UNK>\",\n",
        "                 mask_token=\"<MASK>\", begin_seq_token=\"<BEGIN>\",\n",
        "                 end_seq_token=\"<END>\"):\n",
        "\n",
        "        super(SequenceVocabulary, self).__init__(token_to_idx)\n",
        "\n",
        "        self.mask_token = mask_token\n",
        "        self.unk_token = unk_token\n",
        "        self.begin_seq_token = begin_seq_token\n",
        "        self.end_seq_token = end_seq_token\n",
        "\n",
        "        self.mask_index = self.add_token(self.mask_token)\n",
        "        self.unk_index = self.add_token(self.unk_token)\n",
        "        self.begin_seq_index = self.add_token(self.begin_seq_token)\n",
        "        self.end_seq_index = self.add_token(self.end_seq_token)\n",
        "        \n",
        "        # Index to token\n",
        "        self.idx_to_token = {idx: token \\\n",
        "                             for token, idx in self.token_to_idx.items()}\n",
        "\n",
        "    def to_serializable(self):\n",
        "        contents = super(SequenceVocabulary, self).to_serializable()\n",
        "        contents.update({'unk_token': self.unk_token,\n",
        "                         'mask_token': self.mask_token,\n",
        "                         'begin_seq_token': self.begin_seq_token,\n",
        "                         'end_seq_token': self.end_seq_token})\n",
        "        return contents\n",
        "\n",
        "    def lookup_token(self, token):\n",
        "        return self.token_to_idx.get(token, self.unk_index)\n",
        "    \n",
        "    def lookup_index(self, index):\n",
        "        if index not in self.idx_to_token:\n",
        "            raise KeyError(\"the index (%d) is not in the SequenceVocabulary\" % index)\n",
        "        return self.idx_to_token[index]\n",
        "    \n",
        "    def __str__(self):\n",
        "        return \"<SequenceVocabulary(size=%d)>\" % len(self.token_to_idx)\n",
        "\n",
        "    def __len__(self):\n",
        "        return len(self.token_to_idx)\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "g5UHjpi3El37",
        "colab_type": "code",
        "outputId": "4ca7dd37-112e-4839-becf-6a42ed3716fb",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 85
        }
      },
      "cell_type": "code",
      "source": [
        "# Get word counts\n",
        "word_counts = Counter()\n",
        "for title in split_df.title:\n",
        "    for token in title.split(\" \"):\n",
        "        if token not in string.punctuation:\n",
        "            word_counts[token] += 1\n",
        "\n",
        "# Create SequenceVocabulary instance\n",
        "title_vocab = SequenceVocabulary()\n",
        "for word, word_count in word_counts.items():\n",
        "    if word_count >= args.cutoff:\n",
        "        title_vocab.add_token(word)\n",
        "print (title_vocab) # __str__\n",
        "print (len(title_vocab)) # __len__\n",
        "index = title_vocab.lookup_token(\"general\")\n",
        "print (index)\n",
        "print (title_vocab.lookup_index(index))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<SequenceVocabulary(size=4400)>\n",
            "4400\n",
            "4\n",
            "general\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "4Dag6H0SFHAG",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Vectorizer"
      ]
    },
    {
      "metadata": {
        "id": "VQIfxcUuKwzz",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Something new that we introduce in this Vectorizer is calculating the length of our input sequence. We will use this later on to extract the last relevant hidden state for each input sequence."
      ]
    },
    {
      "metadata": {
        "id": "tsNtEnhBEl6s",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "class NewsVectorizer(object):\n",
        "    def __init__(self, title_vocab, category_vocab):\n",
        "        self.title_vocab = title_vocab\n",
        "        self.category_vocab = category_vocab\n",
        "\n",
        "    def vectorize(self, title):\n",
        "        indices = [self.title_vocab.lookup_token(token) for token in title.split(\" \")]\n",
        "        indices = [self.title_vocab.begin_seq_index] + indices + \\\n",
        "            [self.title_vocab.end_seq_index]\n",
        "        \n",
        "        # Create vector\n",
        "        title_length = len(indices)\n",
        "        vector = np.zeros(title_length, dtype=np.int64)\n",
        "        vector[:len(indices)] = indices\n",
        "\n",
        "        return vector, title_length\n",
        "    \n",
        "    def unvectorize(self, vector):\n",
        "        tokens = [self.title_vocab.lookup_index(index) for index in vector]\n",
        "        title = \" \".join(token for token in tokens)\n",
        "        return title\n",
        "\n",
        "    @classmethod\n",
        "    def from_dataframe(cls, df, cutoff):\n",
        "        \n",
        "        # Create class vocab\n",
        "        category_vocab = Vocabulary()        \n",
        "        for category in sorted(set(df.category)):\n",
        "            category_vocab.add_token(category)\n",
        "\n",
        "        # Get word counts\n",
        "        word_counts = Counter()\n",
        "        for title in df.title:\n",
        "            for token in title.split(\" \"):\n",
        "                word_counts[token] += 1\n",
        "        \n",
        "        # Create title vocab\n",
        "        title_vocab = SequenceVocabulary()\n",
        "        for word, word_count in word_counts.items():\n",
        "            if word_count >= cutoff:\n",
        "                title_vocab.add_token(word)\n",
        "        \n",
        "        return cls(title_vocab, category_vocab)\n",
        "\n",
        "    @classmethod\n",
        "    def from_serializable(cls, contents):\n",
        "        title_vocab = SequenceVocabulary.from_serializable(contents['title_vocab'])\n",
        "        category_vocab = Vocabulary.from_serializable(contents['category_vocab'])\n",
        "        return cls(title_vocab=title_vocab, category_vocab=category_vocab)\n",
        "    \n",
        "    def to_serializable(self):\n",
        "        return {'title_vocab': self.title_vocab.to_serializable(),\n",
        "                'category_vocab': self.category_vocab.to_serializable()}"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "JtRRXU53El9Y",
        "colab_type": "code",
        "outputId": "47ddcba1-de72-45b1-ef72-9cf21050abb5",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 119
        }
      },
      "cell_type": "code",
      "source": [
        "# Vectorizer instance\n",
        "vectorizer = NewsVectorizer.from_dataframe(split_df, cutoff=args.cutoff)\n",
        "print (vectorizer.title_vocab)\n",
        "print (vectorizer.category_vocab)\n",
        "vectorized_title, title_length = vectorizer.vectorize(preprocess_text(\n",
        "    \"Roger Federer wins the Wimbledon tennis tournament.\"))\n",
        "print (np.shape(vectorized_title))\n",
        "print (\"title_length:\", title_length)\n",
        "print (vectorized_title)\n",
        "print (vectorizer.unvectorize(vectorized_title))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<SequenceVocabulary(size=4404)>\n",
            "<Vocabulary(size=4)>\n",
            "(10,)\n",
            "title_length: 10\n",
            "[   2    1 4151 1231   25    1 2392 4076   38    3]\n",
            "<BEGIN> <UNK> federer wins the <UNK> tennis tournament . <END>\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "uk_QvpVfFM0S",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Dataset"
      ]
    },
    {
      "metadata": {
        "id": "oU7oDdelFMR9",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from torch.utils.data import Dataset, DataLoader"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "pB7FHmiSFMXA",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "class NewsDataset(Dataset):\n",
        "    def __init__(self, df, vectorizer):\n",
        "        self.df = df\n",
        "        self.vectorizer = vectorizer\n",
        "\n",
        "        # Data splits\n",
        "        self.train_df = self.df[self.df.split=='train']\n",
        "        self.train_size = len(self.train_df)\n",
        "        self.val_df = self.df[self.df.split=='val']\n",
        "        self.val_size = len(self.val_df)\n",
        "        self.test_df = self.df[self.df.split=='test']\n",
        "        self.test_size = len(self.test_df)\n",
        "        self.lookup_dict = {'train': (self.train_df, self.train_size), \n",
        "                            'val': (self.val_df, self.val_size),\n",
        "                            'test': (self.test_df, self.test_size)}\n",
        "        self.set_split('train')\n",
        "\n",
        "        # Class weights (for imbalances)\n",
        "        class_counts = df.category.value_counts().to_dict()\n",
        "        def sort_key(item):\n",
        "            return self.vectorizer.category_vocab.lookup_token(item[0])\n",
        "        sorted_counts = sorted(class_counts.items(), key=sort_key)\n",
        "        frequencies = [count for _, count in sorted_counts]\n",
        "        self.class_weights = 1.0 / torch.tensor(frequencies, dtype=torch.float32)\n",
        "\n",
        "    @classmethod\n",
        "    def load_dataset_and_make_vectorizer(cls, df, cutoff):\n",
        "        train_df = df[df.split=='train']\n",
        "        return cls(df, NewsVectorizer.from_dataframe(train_df, cutoff))\n",
        "\n",
        "    @classmethod\n",
        "    def load_dataset_and_load_vectorizer(cls, df, vectorizer_filepath):\n",
        "        vectorizer = cls.load_vectorizer_only(vectorizer_filepath)\n",
        "        return cls(df, vectorizer)\n",
        "\n",
        "    def load_vectorizer_only(vectorizer_filepath):\n",
        "        with open(vectorizer_filepath) as fp:\n",
        "            return NewsVectorizer.from_serializable(json.load(fp))\n",
        "\n",
        "    def save_vectorizer(self, vectorizer_filepath):\n",
        "        with open(vectorizer_filepath, \"w\") as fp:\n",
        "            json.dump(self.vectorizer.to_serializable(), fp)\n",
        "\n",
        "    def set_split(self, split=\"train\"):\n",
        "        self.target_split = split\n",
        "        self.target_df, self.target_size = self.lookup_dict[split]\n",
        "\n",
        "    def __str__(self):\n",
        "        return \"<Dataset(split={0}, size={1})\".format(\n",
        "            self.target_split, self.target_size)\n",
        "\n",
        "    def __len__(self):\n",
        "        return self.target_size\n",
        "\n",
        "    def __getitem__(self, index):\n",
        "        row = self.target_df.iloc[index]\n",
        "        title_vector, title_length = self.vectorizer.vectorize(row.title)\n",
        "        category_index = self.vectorizer.category_vocab.lookup_token(row.category)\n",
        "        return {'title': title_vector, 'title_length': title_length, \n",
        "                'category': category_index}\n",
        "\n",
        "    def get_num_batches(self, batch_size):\n",
        "        return len(self) // batch_size\n",
        "\n",
        "    def generate_batches(self, batch_size, collate_fn, shuffle=True, \n",
        "                         drop_last=False, device=\"cpu\"):\n",
        "        dataloader = DataLoader(dataset=self, batch_size=batch_size,\n",
        "                                collate_fn=collate_fn, shuffle=shuffle, \n",
        "                                drop_last=drop_last)\n",
        "        for data_dict in dataloader:\n",
        "            out_data_dict = {}\n",
        "            for name, tensor in data_dict.items():\n",
        "                out_data_dict[name] = data_dict[name].to(device)\n",
        "            yield out_data_dict"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "_Dpb6ZHJFMeb",
        "colab_type": "code",
        "outputId": "c24edb33-310c-428f-94fc-9e53446e080f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 85
        }
      },
      "cell_type": "code",
      "source": [
        "# Dataset instance\n",
        "dataset = NewsDataset.load_dataset_and_make_vectorizer(df=split_df,\n",
        "                                                       cutoff=args.cutoff)\n",
        "print (dataset) # __str__\n",
        "input_ = dataset[5] # __getitem__\n",
        "print (input_['title'], input_['title_length'], input_['category'])\n",
        "print (dataset.vectorizer.unvectorize(input_['title']))\n",
        "print (dataset.class_weights)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<Dataset(split=train, size=84000)\n",
            "[ 2 31 32 10 33 13  3] 7 0\n",
            "<BEGIN> software firm to cut jobs <END>\n",
            "tensor([3.3333e-05, 3.3333e-05, 3.3333e-05, 3.3333e-05])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "_IUIqtbvFUAG",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Model"
      ]
    },
    {
      "metadata": {
        "id": "xJV5WlDiFVVz",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "input → embedding → RNN → FC "
      ]
    },
    {
      "metadata": {
        "id": "rZCzdZZ9FMhm",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import torch.nn as nn\n",
        "import torch.nn.functional as F"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "wbWO4lZcIdqZ",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def gather_last_relevant_hidden(hiddens, x_lengths):\n",
        "    x_lengths = x_lengths.long().detach().cpu().numpy() - 1\n",
        "    out = []\n",
        "    for batch_index, column_index in enumerate(x_lengths):\n",
        "        out.append(hiddens[batch_index, column_index])\n",
        "    return torch.stack(out)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "9TT66Y-UFMcZ",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "class NewsModel(nn.Module):\n",
        "    def __init__(self, embedding_dim, num_embeddings, rnn_hidden_dim, \n",
        "                 hidden_dim, output_dim, num_layers, bidirectional, dropout_p, \n",
        "                 pretrained_embeddings=None, freeze_embeddings=False, \n",
        "                 padding_idx=0):\n",
        "        super(NewsModel, self).__init__()\n",
        "        \n",
        "        if pretrained_embeddings is None:\n",
        "            self.embeddings = nn.Embedding(embedding_dim=embedding_dim,\n",
        "                                          num_embeddings=num_embeddings,\n",
        "                                          padding_idx=padding_idx)\n",
        "        else:\n",
        "            pretrained_embeddings = torch.from_numpy(pretrained_embeddings).float()\n",
        "            self.embeddings = nn.Embedding(embedding_dim=embedding_dim,\n",
        "                                           num_embeddings=num_embeddings,\n",
        "                                           padding_idx=padding_idx,\n",
        "                                           _weight=pretrained_embeddings)\n",
        "        \n",
        "        # Conv weights\n",
        "        self.gru = nn.GRU(input_size=embedding_dim, hidden_size=rnn_hidden_dim, \n",
        "                          num_layers=num_layers, batch_first=True, \n",
        "                          bidirectional=bidirectional)\n",
        "     \n",
        "        # FC weights\n",
        "        self.dropout = nn.Dropout(dropout_p)\n",
        "        self.fc1 = nn.Linear(rnn_hidden_dim, hidden_dim)\n",
        "        self.fc2 = nn.Linear(hidden_dim, output_dim)\n",
        "        \n",
        "        if freeze_embeddings:\n",
        "            self.embeddings.weight.requires_grad = False\n",
        "\n",
        "    def forward(self, x_in, x_lengths, apply_softmax=False):\n",
        "        \n",
        "        # Embed\n",
        "        x_in = self.embeddings(x_in)\n",
        "            \n",
        "        # Feed into RNN\n",
        "        out, h_n = self.gru(x_in)\n",
        "        \n",
        "        # Gather the last relevant hidden state\n",
        "        out = gather_last_relevant_hidden(out, x_lengths)\n",
        "\n",
        "        # FC layers\n",
        "        z = self.dropout(out)\n",
        "        z = self.fc1(z)\n",
        "        z = self.dropout(z)\n",
        "        y_pred = self.fc2(z)\n",
        "\n",
        "        if apply_softmax:\n",
        "            y_pred = F.softmax(y_pred, dim=1)\n",
        "        return y_pred"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "jHPYCPd7Fl3M",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Training"
      ]
    },
    {
      "metadata": {
        "id": "D3seBMA7FlcC",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import torch.optim as optim"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "HnRKWLekFlnM",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "class Trainer(object):\n",
        "    def __init__(self, dataset, model, model_state_file, save_dir, device, shuffle, \n",
        "               num_epochs, batch_size, learning_rate, early_stopping_criteria):\n",
        "        self.dataset = dataset\n",
        "        self.class_weights = dataset.class_weights.to(device)\n",
        "        self.model = model.to(device)\n",
        "        self.save_dir = save_dir\n",
        "        self.device = device\n",
        "        self.shuffle = shuffle\n",
        "        self.num_epochs = num_epochs\n",
        "        self.batch_size = batch_size\n",
        "        self.loss_func = nn.CrossEntropyLoss(self.class_weights)\n",
        "        self.optimizer = optim.Adam(self.model.parameters(), lr=learning_rate)\n",
        "        self.scheduler = optim.lr_scheduler.ReduceLROnPlateau(\n",
        "            optimizer=self.optimizer, mode='min', factor=0.5, patience=1)\n",
        "        self.train_state = {\n",
        "            'done_training': False,\n",
        "            'stop_early': False, \n",
        "            'early_stopping_step': 0,\n",
        "            'early_stopping_best_val': 1e8,\n",
        "            'early_stopping_criteria': early_stopping_criteria,\n",
        "            'learning_rate': learning_rate,\n",
        "            'epoch_index': 0,\n",
        "            'train_loss': [],\n",
        "            'train_acc': [],\n",
        "            'val_loss': [],\n",
        "            'val_acc': [],\n",
        "            'test_loss': -1,\n",
        "            'test_acc': -1,\n",
        "            'model_filename': model_state_file}\n",
        "    \n",
        "    def update_train_state(self):\n",
        "\n",
        "        # Verbose\n",
        "        print (\"[EPOCH]: {0} | [LR]: {1} | [TRAIN LOSS]: {2:.2f} | [TRAIN ACC]: {3:.1f}% | [VAL LOSS]: {4:.2f} | [VAL ACC]: {5:.1f}%\".format(\n",
        "          self.train_state['epoch_index'], self.train_state['learning_rate'], \n",
        "            self.train_state['train_loss'][-1], self.train_state['train_acc'][-1], \n",
        "            self.train_state['val_loss'][-1], self.train_state['val_acc'][-1]))\n",
        "\n",
        "        # Save one model at least\n",
        "        if self.train_state['epoch_index'] == 0:\n",
        "            torch.save(self.model.state_dict(), self.train_state['model_filename'])\n",
        "            self.train_state['stop_early'] = False\n",
        "\n",
        "        # Save model if performance improved\n",
        "        elif self.train_state['epoch_index'] >= 1:\n",
        "            loss_tm1, loss_t = self.train_state['val_loss'][-2:]\n",
        "\n",
        "            # If loss worsened\n",
        "            if loss_t >= self.train_state['early_stopping_best_val']:\n",
        "                # Update step\n",
        "                self.train_state['early_stopping_step'] += 1\n",
        "\n",
        "            # Loss decreased\n",
        "            else:\n",
        "                # Save the best model\n",
        "                if loss_t < self.train_state['early_stopping_best_val']:\n",
        "                    torch.save(self.model.state_dict(), self.train_state['model_filename'])\n",
        "\n",
        "                # Reset early stopping step\n",
        "                self.train_state['early_stopping_step'] = 0\n",
        "\n",
        "            # Stop early ?\n",
        "            self.train_state['stop_early'] = self.train_state['early_stopping_step'] \\\n",
        "              >= self.train_state['early_stopping_criteria']\n",
        "        return self.train_state\n",
        "  \n",
        "    def compute_accuracy(self, y_pred, y_target):\n",
        "        _, y_pred_indices = y_pred.max(dim=1)\n",
        "        n_correct = torch.eq(y_pred_indices, y_target).sum().item()\n",
        "        return n_correct / len(y_pred_indices) * 100\n",
        "    \n",
        "    def pad_seq(self, seq, length):\n",
        "        vector = np.zeros(length, dtype=np.int64)\n",
        "        vector[:len(seq)] = seq\n",
        "        vector[len(seq):] = self.dataset.vectorizer.title_vocab.mask_index\n",
        "        return vector\n",
        "    \n",
        "    def collate_fn(self, batch):\n",
        "        \n",
        "        # Make a deep copy\n",
        "        batch_copy = copy.deepcopy(batch)\n",
        "        processed_batch = {\"title\": [], \"title_length\": [], \"category\": []}\n",
        "        \n",
        "        # Get max sequence length\n",
        "        get_length = lambda sample: len(sample[\"title\"])\n",
        "        max_seq_length = max(map(get_length, batch))\n",
        "        \n",
        "        # Pad\n",
        "        for i, sample in enumerate(batch_copy):\n",
        "            padded_seq = self.pad_seq(sample[\"title\"], max_seq_length)\n",
        "            processed_batch[\"title\"].append(padded_seq)\n",
        "            processed_batch[\"title_length\"].append(sample[\"title_length\"])\n",
        "            processed_batch[\"category\"].append(sample[\"category\"])\n",
        "            \n",
        "        # Convert to appropriate tensor types\n",
        "        processed_batch[\"title\"] = torch.LongTensor(\n",
        "            processed_batch[\"title\"])\n",
        "        processed_batch[\"title_length\"] = torch.LongTensor(\n",
        "            processed_batch[\"title_length\"])\n",
        "        processed_batch[\"category\"] = torch.LongTensor(\n",
        "            processed_batch[\"category\"])\n",
        "        \n",
        "        return processed_batch   \n",
        "  \n",
        "    def run_train_loop(self):\n",
        "        for epoch_index in range(self.num_epochs):\n",
        "            self.train_state['epoch_index'] = epoch_index\n",
        "      \n",
        "            # Iterate over train dataset\n",
        "\n",
        "            # initialize batch generator, set loss and acc to 0, set train mode on\n",
        "            self.dataset.set_split('train')\n",
        "            batch_generator = self.dataset.generate_batches(\n",
        "                batch_size=self.batch_size, collate_fn=self.collate_fn, \n",
        "                shuffle=self.shuffle, device=self.device)\n",
        "            running_loss = 0.0\n",
        "            running_acc = 0.0\n",
        "            self.model.train()\n",
        "\n",
        "            for batch_index, batch_dict in enumerate(batch_generator):\n",
        "                # zero the gradients\n",
        "                self.optimizer.zero_grad()\n",
        "\n",
        "                # compute the output\n",
        "                y_pred = self.model(batch_dict['title'], batch_dict['title_length'])\n",
        "\n",
        "                # compute the loss\n",
        "                loss = self.loss_func(y_pred, batch_dict['category'])\n",
        "                loss_t = loss.item()\n",
        "                running_loss += (loss_t - running_loss) / (batch_index + 1)\n",
        "\n",
        "                # compute gradients using loss\n",
        "                loss.backward()\n",
        "\n",
        "                # use optimizer to take a gradient step\n",
        "                self.optimizer.step()\n",
        "                \n",
        "                # compute the accuracy\n",
        "                acc_t = self.compute_accuracy(y_pred, batch_dict['category'])\n",
        "                running_acc += (acc_t - running_acc) / (batch_index + 1)\n",
        "\n",
        "            self.train_state['train_loss'].append(running_loss)\n",
        "            self.train_state['train_acc'].append(running_acc)\n",
        "\n",
        "            # Iterate over val dataset\n",
        "\n",
        "            # # initialize batch generator, set loss and acc to 0; set eval mode on\n",
        "            self.dataset.set_split('val')\n",
        "            batch_generator = self.dataset.generate_batches(\n",
        "                batch_size=self.batch_size, collate_fn=self.collate_fn, \n",
        "                shuffle=self.shuffle, device=self.device)\n",
        "            running_loss = 0.\n",
        "            running_acc = 0.\n",
        "            self.model.eval()\n",
        "\n",
        "            for batch_index, batch_dict in enumerate(batch_generator):\n",
        "\n",
        "                # compute the output\n",
        "                y_pred =  self.model(batch_dict['title'], batch_dict['title_length'])\n",
        "\n",
        "                # compute the loss\n",
        "                loss = self.loss_func(y_pred, batch_dict['category'])\n",
        "                loss_t = loss.to(\"cpu\").item()\n",
        "                running_loss += (loss_t - running_loss) / (batch_index + 1)\n",
        "\n",
        "                # compute the accuracy\n",
        "                acc_t = self.compute_accuracy(y_pred, batch_dict['category'])\n",
        "                running_acc += (acc_t - running_acc) / (batch_index + 1)\n",
        "\n",
        "            self.train_state['val_loss'].append(running_loss)\n",
        "            self.train_state['val_acc'].append(running_acc)\n",
        "\n",
        "            self.train_state = self.update_train_state()\n",
        "            self.scheduler.step(self.train_state['val_loss'][-1])\n",
        "            if self.train_state['stop_early']:\n",
        "                break\n",
        "          \n",
        "    def run_test_loop(self):\n",
        "        # initialize batch generator, set loss and acc to 0; set eval mode on\n",
        "        self.dataset.set_split('test')\n",
        "        batch_generator = self.dataset.generate_batches(\n",
        "            batch_size=self.batch_size, collate_fn=self.collate_fn, \n",
        "            shuffle=self.shuffle, device=self.device)\n",
        "        running_loss = 0.0\n",
        "        running_acc = 0.0\n",
        "        self.model.eval()\n",
        "\n",
        "        for batch_index, batch_dict in enumerate(batch_generator):\n",
        "            # compute the output\n",
        "            y_pred =  self.model(batch_dict['title'], batch_dict['title_length'])\n",
        "\n",
        "            # compute the loss\n",
        "            loss = self.loss_func(y_pred, batch_dict['category'])\n",
        "            loss_t = loss.item()\n",
        "            running_loss += (loss_t - running_loss) / (batch_index + 1)\n",
        "\n",
        "            # compute the accuracy\n",
        "            acc_t = self.compute_accuracy(y_pred, batch_dict['category'])\n",
        "            running_acc += (acc_t - running_acc) / (batch_index + 1)\n",
        "\n",
        "        self.train_state['test_loss'] = running_loss\n",
        "        self.train_state['test_acc'] = running_acc\n",
        "    \n",
        "    def plot_performance(self):\n",
        "        # Figure size\n",
        "        plt.figure(figsize=(15,5))\n",
        "\n",
        "        # Plot Loss\n",
        "        plt.subplot(1, 2, 1)\n",
        "        plt.title(\"Loss\")\n",
        "        plt.plot(trainer.train_state[\"train_loss\"], label=\"train\")\n",
        "        plt.plot(trainer.train_state[\"val_loss\"], label=\"val\")\n",
        "        plt.legend(loc='upper right')\n",
        "\n",
        "        # Plot Accuracy\n",
        "        plt.subplot(1, 2, 2)\n",
        "        plt.title(\"Accuracy\")\n",
        "        plt.plot(trainer.train_state[\"train_acc\"], label=\"train\")\n",
        "        plt.plot(trainer.train_state[\"val_acc\"], label=\"val\")\n",
        "        plt.legend(loc='lower right')\n",
        "\n",
        "        # Save figure\n",
        "        plt.savefig(os.path.join(self.save_dir, \"performance.png\"))\n",
        "\n",
        "        # Show plots\n",
        "        plt.show()\n",
        "    \n",
        "    def save_train_state(self):\n",
        "        self.train_state[\"done_training\"] = True\n",
        "        with open(os.path.join(self.save_dir, \"train_state.json\"), \"w\") as fp:\n",
        "            json.dump(self.train_state, fp)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "ICkiOaGtFlk-",
        "colab_type": "code",
        "outputId": "3e2b70e0-a724-4953-c803-ac93a8a8b687",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 136
        }
      },
      "cell_type": "code",
      "source": [
        "# Initialization\n",
        "dataset = NewsDataset.load_dataset_and_make_vectorizer(df=split_df,\n",
        "                                                       cutoff=args.cutoff)\n",
        "dataset.save_vectorizer(args.vectorizer_file)\n",
        "vectorizer = dataset.vectorizer\n",
        "model = NewsModel(embedding_dim=args.embedding_dim, \n",
        "                  num_embeddings=len(vectorizer.title_vocab), \n",
        "                  rnn_hidden_dim=args.rnn_hidden_dim,\n",
        "                  hidden_dim=args.hidden_dim,\n",
        "                  output_dim=len(vectorizer.category_vocab),\n",
        "                  num_layers=args.num_layers,\n",
        "                  bidirectional=args.bidirectional,\n",
        "                  dropout_p=args.dropout_p, \n",
        "                  pretrained_embeddings=None, \n",
        "                  padding_idx=vectorizer.title_vocab.mask_index)\n",
        "print (model.named_modules)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<bound method Module.named_modules of NewsModel(\n",
            "  (embeddings): Embedding(3406, 100, padding_idx=0)\n",
            "  (gru): GRU(100, 128, batch_first=True)\n",
            "  (dropout): Dropout(p=0.1)\n",
            "  (fc1): Linear(in_features=128, out_features=100, bias=True)\n",
            "  (fc2): Linear(in_features=100, out_features=4, bias=True)\n",
            ")>\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "tuaRZ4DiFlh1",
        "colab_type": "code",
        "outputId": "5a0c146d-cf5d-4d05-ed4e-ece620c1b33e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 102
        }
      },
      "cell_type": "code",
      "source": [
        "# Train\n",
        "trainer = Trainer(dataset=dataset, model=model, \n",
        "                  model_state_file=args.model_state_file, \n",
        "                  save_dir=args.save_dir, device=args.device,\n",
        "                  shuffle=args.shuffle, num_epochs=args.num_epochs, \n",
        "                  batch_size=args.batch_size, learning_rate=args.learning_rate, \n",
        "                  early_stopping_criteria=args.early_stopping_criteria)\n",
        "trainer.run_train_loop()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[EPOCH]: 0 | [LR]: 0.001 | [TRAIN LOSS]: 0.75 | [TRAIN ACC]: 70.1% | [VAL LOSS]: 0.54 | [VAL ACC]: 80.4%\n",
            "[EPOCH]: 1 | [LR]: 0.001 | [TRAIN LOSS]: 0.48 | [TRAIN ACC]: 82.7% | [VAL LOSS]: 0.48 | [VAL ACC]: 82.7%\n",
            "[EPOCH]: 2 | [LR]: 0.001 | [TRAIN LOSS]: 0.41 | [TRAIN ACC]: 85.1% | [VAL LOSS]: 0.47 | [VAL ACC]: 83.1%\n",
            "[EPOCH]: 3 | [LR]: 0.001 | [TRAIN LOSS]: 0.36 | [TRAIN ACC]: 86.6% | [VAL LOSS]: 0.47 | [VAL ACC]: 83.6%\n",
            "[EPOCH]: 4 | [LR]: 0.001 | [TRAIN LOSS]: 0.32 | [TRAIN ACC]: 88.3% | [VAL LOSS]: 0.48 | [VAL ACC]: 83.4%\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "mzRJIz88Flfe",
        "colab_type": "code",
        "outputId": "0c47971c-838c-4c04-ee63-0f71c3264c49",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 335
        }
      },
      "cell_type": "code",
      "source": [
        "# Plot performance\n",
        "trainer.plot_performance()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAA2gAAAE+CAYAAAD4XjP+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzs/Xd0XPWB//8/p6qXUe+yLRe5W+6W\n3MG40vsGQpbk7AlLsr+zYVnzUbIhsLScb5xPgGxI+bBLQrKBADYOmGqwjbtsSzZukrskS1azep2i\n+f0hMdjYuGBJVyO9Huf4MHPv3JmXxkaal97v+74mr9frRURERERERAxnNjqAiIiIiIiIdFFBExER\nERER6SdU0ERERERERPoJFTQREREREZF+QgVNRERERESkn1BBExERERER6SdU0ES+oVGjRlFRUWF0\nDBERkT5xzz33cNNNNxkdQ2TAU0ETERERkUs6cuQIYWFhJCUlUVBQYHQckQFNBU2kh3V0dPDTn/6U\nxYsXs3TpUp577jk8Hg8Af/7zn1m6dClLlizhjjvu4OjRo5fcLiIi0h+sWbOGJUuWsGLFCt5++23f\n9rfffpvFixezePFiHn30UZxO59du37lzJ4sWLfIde+79F198kZ/85CfccccdvPLKK3R2dvLEE0+w\nePFiFi5cyKOPPorL5QKgtraW73//+1x33XXceOONbNmyhY0bN7JixYrzMt92222sX7++t98akR5n\nNTqAyEDzxz/+kYqKCtatW4fb7ea+++7j3Xff5brrruP5559nw4YNhIaG8v7777Nx40YSExMvun3E\niBFGfykiIiJ4PB4+/vhjHn74YSwWC6tWrcLpdFJVVcXPf/5z3n77beLi4vjhD3/In/70J5YsWXLR\n7ePHj7/k62zatIm1a9cSFRXFhx9+yO7du3n33Xfp7Ozk1ltv5b333uPmm29m1apVZGRk8Nvf/pZD\nhw7xj//4j2zevJnq6moKCwvJzMykvLyckpIS5s6d20fvkkjPUUET6WEbN27kwQcfxGq1YrVaufHG\nG9m6dSvLli3DZDLx5ptvsmLFCpYuXQqAy+W66HYREZH+YMuWLYwfP57Q0FAApk+fzoYNG6ivrycr\nK4v4+HgAVq1ahcVi4a233rro9j179lzydSZOnEhUVBQAixcvZsGCBdhsNgDGjx9PaWkp0FXk/vCH\nPwAwZswYPvnkE+x2O4sXL2bdunVkZmayfv16rrvuOux2e8+/ISK9TFMcRXpYbW0tERERvvsRERGc\nPXsWm83GK6+8Qn5+PosXL+Yf/uEfKCoq+trtIiIi/cHq1avZuHEjU6dOZerUqXz00UesWbOGuro6\nwsPDfY8LCAjAarV+7fbLOfdnZ21tLStXrmTx4sUsWbKETz75BK/XC0B9fT1hYWG+x35RHJcvX866\ndesAWL9+PcuWLbu2L1zEICpoIj0sJiaG+vp63/36+npiYmKArt/0vfDCC2zfvp3Zs2fz+OOPX3K7\niIiIkRoaGsjLy2Pnzp3s3r2b3bt3s2vXLvbv34/ZbKaurs732ObmZmpqanA4HBfdbrFYfOdkAzQ2\nNn7t6/7f//t/sVqtvPPOO3zwwQfMmzfPty8yMvK85z99+jQul4tp06bhdrvZsGEDR48eJTs7u6fe\nBpE+pYIm0sPmz5/Pm2++icfjobW1lbVr1zJv3jyKior4l3/5F5xOJ3a7nXHjxmEymb52u4iIiNHW\nrVvHzJkzz5sqaLVamT17Nk6nk/z8fE6fPo3X6+Xxxx/nzTffZN68eRfdHhsbS3V1NWfPnsXj8fDO\nO+987euePXuWkSNHYrfbKSwspKCggNbWVgAWLlzImjVrADh27Bi33XYbHo8Hs9nMsmXL+M///E8W\nLlzomx4p4m90DprINbj//vuxWCy++0899RT3338/paWlLF++HJPJxJIlS3znlaWkpLBixQpsNhsh\nISH89Kc/ZeTIkRfdLiIiYrS3336bBx544ILtixYt4je/+Q1PPvkkDzzwABaLhfHjx/OP//iPBAQE\nfO3222+/nVtuuYWkpCRuvvlmDh8+fNHXffDBB1m5ciWrV69m6tSprFy5kh//+MdMmDCBRx99lJUr\nV7Jw4UJCQkL4xS9+QWBgINA1zfF//ud/NL1R/JrJ+8WEXhERERERP1ZTU8Ott97Kxo0bz/sFqog/\n0RRHERERERkQXnjhBe69916VM/FrKmgiIiIi4tdqamq47rrrqKmp4cEHHzQ6jsg10RRHERERERGR\nfkIjaCIiIiIiIv2ECpqIiIiIiEg/0efL7FdXN13zczgcwdTVtfZAmr7hT3mVtXf4U1bwr7zK2jt6\nKmtsbFgPpBk8BtvPSH/KCv6VV1l7h7L2Hn/K2xNZL/Xz0S9H0KxW/1qZx5/yKmvv8Kes4F95lbV3\n+FNWOZ8//d35U1bwr7zK2juUtff4U97ezuqXBU1ERERERGQgUkETERERERHpJ1TQRERERERE+gkV\nNBERERERkX5CBU1ERERERKSfUEETERERERHpJ1TQRERERERE+gkVNBGRQWTjxk+u6HFPP/005eVl\nvZxGREREvkoFTURkkDhzppz16z+8osf++Mc/JikpuZcTiYiIyFdZjQ5wtTqcHt7ZfIJJwxwE2v0u\nvoiIYX75y59z+PBB5syZxg03LOXMmXJ+9avf8OyzT1JdXUVbWxsPPvhP5OTM4f777+cHP/gRGzZ8\nQktLMyUlxZSVneZf/uURZs3KMfpLERER6VOdXi+llc0cPV3PlLGJOIJ6r4f4XcMpKq3j92/v57op\nKXxr0Uij44iI+I17772f1av/xtChGZSUnOI3v/l/1NXVMn36TJYuXUFZ2Wn+4z8eIydnznnHVVVV\n8otfvMCOHdtYu/YtFTQRERnwvF4v5WdbKSyu43BxHUUldbS0uwE4VdXM95aN7rXX9ruCNmZIFInR\nIWwsKOP6qSnEO4KNjiQictX+9ukxdhVW9ehzTsuM466Fw6/osaNHjwUgLCycw4cP8ve/r8ZkMtPY\n2HDBYydMmARAXFwczc3NPRdYRESkn/B6vVTVt3G4uI7C4joKS+ppbHH69keHB5I1IpbM9EiunzmU\n1ub2XsvidwXNajHz7eWj+fmfdrN60wkeumWc0ZFERPyOzWYD4OOPP6CxsZH/+q//R2NjI9/73v0X\nPNZisfhue73ePssoIiLSm842tHcVspKuUbK6pg7fvohQOzPHxpOZ5mB0uoPYyCDfvpAgmwraV+VM\nSGJoYji7Cqu4obyBjKQIoyOJiFyVuxYOv+LRrp5iNpvxeDznbauvrycxMQmz2cymTZ/icrn6NJOI\niEhfqW/u6B4d6ypk1fVflqzQIBtTR8UyOt1BZrqDhKhgTCaTITn9sqCZTCbuWpDBz/+3gDc2HGfl\nP2QZ9gaKiPiL9PShFBUVkpiYRGRkJADz5y/kscd+xKFDB1i+/Cbi4uL4n//5g8FJRURErl1zm6vr\nHLKSrmmLZ862+vYFBViZNDzGV8iSY0Mw95M+4ZcFDWBUmoNJw2PYe6yGfcfOMmlEjNGRRET6NYfD\nwerV687blpiYxB//+Jrv/g03LAUgNjaM6uomhg37cpRv2LDh/PrXv++bsCIiIleptd3NkdJ637TF\n0qovz5sOsFkYNyyqq5ClOUiPD8Ns7h+F7Kv8tqAB3D4/g33Ha3hz03HGZ0RhMeuybiIiIiIig0GH\n08PR0/W+EbJTFU18caq01WLuLmORjE6PYkhiGFaLf3QFvy5oyTEhzJmQxGf7ytm6v4K5E5OMjiQi\nIgNQS0sLK1eupKGhAZfLxcMPP8zvf//laGJVVRW33nor3//+933bXnzxRd555x3i4+MBuOmmm7jz\nzjv7PLuIyEDhcns4Vtbom7Z4srwRT2dXI7OYTWQkRzA6rWvK4vDkcGxWy2WesX/y64IGcPPsoew4\nWMGazSeYMTqeALt//kWIiEj/tWbNGoYOHcojjzxCZWUlDzzwAB988IFv//e+9z1uvvnmC4779re/\nzX333deXUUVEBgy3p5NTZ5o4XFzL4eI6jpU14vZ0AmAywZCEMDLTHYxOczA8JYJAu99XG2AAFDRH\nWAA3TE/j3W2n+GhXCTfmDDU6koiIDDAOh4OioiIAGhsbcTgcvn3btm1jyJAhJCYmGhVPRGRA6Oz0\nUlzZ5BshO1raQIfry9WHU+NCfeeQjUyNJDjQ76vMRQ2Ir2rpjDQ27S3jvZ0lzJuUTHiI3ehIIiIy\ngCxfvpzVq1ezaNEiGhsb+d3vfufb96c//Ync3NyLHvfBBx/wySefYLfb+clPfkJqampfRRYR6fc6\nvV7Kqls4XFzHyYomPj9WQ1uH27c/MTrYN0I2Ki2SsODB8Rl/QBS0oAArN+UM5S8fH+Gdraf41g0j\njY4kIiIDyNq1a0lKSuLll1+msLCQ3NxcVq9eTWVlJa2traSlpV1wzLx585g5cybTpk1j3bp1PPXU\nU+cVu4txOIKx9sA5E7GxYdf8HH3Fn7KCf+VV1t6hrN+c1+vldFUz+4/X8PnRGvYfr6Gxxenbnxgd\nwpxJyUwYHsP44TFEhQcamPbSevO9HRAFDWDepCQ+3l3Kxr1lXD81hfioYKMjiYj4pTvuuJH33lt3\n+QcOIvn5+cyePRuAzMxMqqqq8Hg8bNq0iZkzZ170mAkTJvhuL1y4kF/84heXfZ26utbLPuZyvrhE\ngj/wp6zgX3mVtXco69Wrrm/rWva+e9piQ/OXhcwRFkD2uAQy0xzkZKVg8nw5ndHT4aK62mVE5Mvq\niff2UgVvwBQ0q8XMHfMy+M3bB3hr03H++dbxRkcSEZEBIj09nX379rF48WLKysoICQnBYrGwf/9+\nFixYcNFjnnrqKZYsWcLUqVPJy8tjxIgRfZxaRKTv1Ta2U1hSR2Fx1/XIzja2+/aFB9uYPjqua9pi\nuoO4yCBM3ReHjo0K7heFsj8YMAUNYMqoWIYlhbO7qJrjZQ1kJEcYHUlEpN948MFv8cwzq0hISKCi\n4gz/5/88QmxsHG1tbbS3t/Ov//ooY8aMMzpmv3T33XeTm5vLfffdh9vt5mc/+xkA1dXVREdH+x5X\nXV3Niy++yJNPPsmdd97J448/jtVqxWQy8dRTTxmUXkSk9zS2OLsLWR2Hi+uorGvz7QsJtDJ5ZGzX\nwh7pDpKig32FTL7egCpoJpOJuxYM57m/5PPGhmOs/NZk/SMQEek2d+4Ctm79jNtvv4vNmzcxd+4C\nMjJGMHfufPbs2cVf/vJHnn76/zM6Zr8UEhLC888/f8H23/72t+fdj42N5cknnwRg1KhRvPbaa32S\nT0Skr7S0uygqqfdNWyyrafHtC7RbmJAR7VtpMTU+FLM+i1+1AVXQAEamRpI1IoaCozXsPVZD1ohY\noyOJiFxg9bF3Kaja36PPmRU3ntuGr/ja/XPnLuDXv/4Vt99+F1u2bOIHP/hXXnvtVf7611dxuVwE\nBvbfk7FFRMQYbR1ujp7+opDVU1LZhLd7n91qZuyQrtGxzHQHQxLCsJjNhuYdCAZcQQO4fV4Ge4/V\n8ObG40zIiNY/FBERYNiwDM6eraaysoKmpiY2b95ITEwc//Ef/0lh4SF+/etfGR1RREQM1uHycKys\ngcLuEbKTZ5ro9HZVMqvFxMjUSN85ZEMTw7FZ9Tm7pw3IgpYUE8LciUls2lvO5s/PMH9SstGRRETO\nc9vwFZcc7eots2bN5ve//w1z5syjvr6OjIyuhSs2bdqA2+2+zNEiIjLQuNydnChvoLB72uKJ8gbc\nnq5CZjaZGJoY5itkGckRBNiu/VIgcmkDsqAB3Dx7KNsPVrB280lmjUkgwK5/TCIi8+Yt4Pvff5BX\nXvkr7e1tPPXU42zYsJ7bb7+L9es/Yt26vxsdUUREepGns5NTZ5ooLOla1OPY6Qac7k4ATEBafFj3\noh6RjEiJJChgwNaFfmvAvuORoQEsmZ7G37ee4sNdJdyUM9ToSCIihhs9eiybNu303f/LX9703Z49\nex4Ay5ffREhICK2tWu5YRMTfdXq9lFY2d51DVlLHkdJ62p1fXm8sOTaE0Wld55CNSoskJNBmYFqB\nAVzQABZPT2NjQRnv7yxh3qRkIkLsRkcSEREREek1nV4v5dUt7CisZvehCopK6mhp/3IKe3xUMDPT\nus4jy0xzEK7Px/3OgC5oQQFWbpo9lD9/dIS/bz3J/TeMMjqSiIiIiEiP6ez0UlLVRFFJPUdKu/6c\nW8iiwwPJGvHltcgcYQEGppUrMaALGsDciUl8vKuUTQXlXD8lhcToEKMjiYiIiIh8I25PJ8UVTRSV\n1lNUUs+xsnraOr6cshgdHsjE4TFMGZNASlQQsZFBBqaVb2LAFzSrxcwd8zP4rzUHWL3pBA/fNt7o\nSCIiIiIiV8Tl9nCivJEjpfUUldZzrKwBp6vTtz/eEcS0zEhGpnb9iYnoKmSxsWFUV+tcYn804Asa\nwOSRsWQkh7PnSDXHyhoYnhxhdCQRERERkQt0uDwcL2vwTVk8Xt6I2/NlIUuKCWFUaiSj0rpWWdSU\nxYFnUBQ0k8nEXQuG8+yf8/nbhmP8n29NxmQyGR1LRERERAa5tg43R083dI+Q1XHqTBOezq7rkJmA\n1LhQRqZFMio1khGpkYQHa1GPgW5QFDSAESmRZI2IoeBoDQVHa5g8MtboSCIiIiIyyLS0u7rKWPcI\nWXFlE96uPobZZCI9IZRRqQ5GpkUyIiVCy94PQldU0J555hn27duHyWQiNzeXCRMmAFBZWcm//du/\n+R5XWlrKI488wo033tg7aa/RHfMz2HfsLG9uPM7E4dFYzGajI4mIiIjIANbY4vSdP1ZUUk9ZdTPd\nfQyL2URGckTXlMXUSDKSI3RhaLl8QcvLy6O4uJjXX3+d48ePk5uby+uvvw5AfHw8r776KgBut5v7\n77+fhQsX9m7ia5AYHcLcSUlsLChj874zzM9KNjqSiIiIiAwgdU0dFJXWcaSkq5SdOdvq22ezmhmV\n1rWYx6g0B8OSwgmwWQxMK/3RZQva9u3buf766wHIyMigoaGB5uZmQkNDz3vcmjVrWLx4MSEh/XsZ\n+5tzhrD9QAVvbznJzLHxBNr1WwoRERER+WZq6tu6RsdK6zlSUk9VfZtvX4DNwtihUV2FLDWSoYnh\n2KyawSWXdtl2UlNTw9ixY333o6KiqK6uvqCgvfHGG/z3f//3ZV/Q4QjGar323xTExoZ94+NuWzCc\nv35UxJYDldy7OPOas1zp6/oLZe0d/pQV/CuvsvYOf8oqItIXvF4vlbWtvumKR0rrONvY4dsfFGBl\nQka0b5QsPT4Mq0WFTK7OVQ8feb84i/EcBQUFDBs27ILSdjF1da2XfczlXOt1HWaPjWfd1pO8teEY\n00bGEBHau8uT+tN1KJS1d/hTVvCvvMraO3oqq0qeiPgzr9dLeU3Ledcgqz2nkIUG2Zg8MtY3QpYa\nF4rZrJXC5dpctqDFxcVRU1Pju19VVUVs7PkrIG7cuJFZs2b1fLpeEhRg5ebZQ3n1wyLWbj3FtxeP\nMjqSiIiIiBiss9PL6epm3wqLRaX1NLe5fPsjwwKYlhnnGyFLignBrEs3SQ+7bEHLycnhxRdf5J57\n7uHgwYPExcVdMFK2f/9+li1b1mshe8OcCYl8vKuUz/aWs2hqConR/fvcORERERHpWZ7OTkoquwpZ\nUUkdR0830Nrh9u2PCg9g1rB436Ie40bGUVPTbGBiGQwuW9AmT57M2LFjueeeezCZTDz++OOsXr2a\nsLAwFi1aBEB1dTXR0dG9HrYnWS1mbp+XwX+t2c9bm07wg9vGGx1JRERERHqR29PJyTONvuuQHS1r\noMPp8e2PjQxk8shY3whZTEQgpnNGyEwaLZM+cEXnoJ17rTOAzMzzF9Z45513ei5RH5o8MobhyRHk\nH6nm6Ol6RqREGh1JRERERHqI0+XhRHlj96IedZwob8Tp7vTtT4wO9p0/NjI1kqjwQAPTinQZ1GvM\nm0wm7lownGf+vIe/bThG7n1T9JsRERERET/V7nRzrKzBN0J28kwjbs+XC9ylxIYwKtXBqLRIRqRG\nEhFiNzCtyMUN6oIGMDwlgikjY9lzpJr8IzVMGRV7+YNERERExHCt7S6Onm7wLXtfXNFEZ/eK4yYT\npMWHMap7hGxEaiShQTaDE4tc3qAvaAC3zRtGwdEa3tx0nInDo3W9ChEREZF+qLnNdc4Ki3WUVjbz\nxfiYxWxiaGIYI9MiGZXqYHhyBMGB+qgr/kf/aoHE6BDmTUpiQ0EZm/eVs2ByitGRRERERAa9huaO\nrtGx0q5SVlbd4ttntZgZ0X3u2Ki0SIYnRRBgtxiYVqRnqKB1u2n2ULYdqGDtlpPMHJtAUIDeGhER\n6dLS0sLKlStpaGjA5XLx8MMP8/vf/57W1laCg4MBWLlyJePGjfMd43K5eOyxxygvL8disfDss8+S\nmppq1Jcg4hdqG9u7lrzvLmWVta2+fXarmdHpXeePjUqNZFhSODarCpkMPGoh3SJC7CydkcbbW07y\nYV4Jt8wZZnQkERHpJ9asWcPQoUN55JFHqKys5IEHHiA2NpZnn32WkSNHXvSYd999l/DwcFatWsWW\nLVtYtWoVv/rVr/o4uUj/VtvYzr6Ttew+WEFRaT01De2+fYF2C+OHRTMyNYJRaQ6GJITpNBQZFFTQ\nznHD9FQ2FJTxYV4p87OSiQwNMDqSiIj0Aw6Hg6KiIgAaGxtxOByXPWb79u3ccsstAGRnZ5Obm9ur\nGUX8RbvTzZ6iarYdqKCwuM53DllIoJVJw2N81yBLiw/FYlYhk8FHBe0cgXYrN88eyp8+LOLvW07y\n7SWZlz9IREQGvOXLl7N69WoWLVpEY2Mjv/vd71i1ahUvvPACdXV1ZGRkkJubS2Dgl9dQqqmpISoq\nCgCz2YzJZMLpdGK3a1lvGXw6vV6KiuvYeqCCPUXVdLi6Lg49IiWC+VNTSYkKJjk2BLMudySigvZV\ncyYm8tGuUj7bd4brp6aSFBNidCQRETHY2rVrSUpK4uWXX6awsJDc3FweeughRo0aRVpaGo8//jh/\n+ctf+O53v/u1z+H1er923xccjmCsPXBOTWxs2DU/R1/xp6zgX3n7Q9ay6mY+3V3Khj2lVNe1ARAf\nFczCqaksmJJKoh9+zuoP7+uV8qes4F95ezOrCtpXWMxm7pyfwYur9/PWpuP88PYJRkcSERGD5efn\nM3v2bAAyMzOpqqpi4cKFWCxdZWrhwoW899575x0TFxdHdXU1mZmZuFwuvF7vZUfP6upaL7n/SsTG\nhlFd3XTNz9MX/Ckr+FdeI7O2tLvIO1zFtv1nOF7eCHSdTzZ7QiI54xIYkRrZNVLm7aS6uknvay/x\np6zgX3l7IuulCp4K2kVMGhHDiJQICo7WcKS0npGpkUZHEhERA6Wnp7Nv3z4WL15MWVkZwcHBfPe7\n3+WFF14gPDycnTt3MmLEiPOOycnJ4YMPPmDOnDls2LCBGTNmGJRepPe5PZ0cOFnLtgMV7D1ag9vT\niQkYO8RB9vhEJo+MJcCmFRdFroQK2kWYTCbuXDCcZ17dwxsbjpF7/xRMmhMtIjJo3X333eTm5nLf\nfffhdrt54oknqKur4zvf+Q5BQUHEx8fzwx/+EICHHnqIl156iWXLlrFt2zbuvfde7HY7zz33nMFf\nhUjPK6lsYtuBCnYcrKCx1QVAYnQwOeMTmTU2AUeYFlwTuVoqaF9jeHIEU0bFsqeomj1F1UzNjDM6\nkoiIGCQkJITnn3/+gu3Lli27YNtLL70E4Lv2mchA09DiZMfBCrYdqKC0qhnoWoHxuskpZI9PYEhC\nmH6xLXINVNAu4fZ5Gew9WsObm44zaUSMrr0hIiIig5LL7WHvsbNs3X+GAydq6fR6sZhNZI2IIXtc\nIhOHR+tzkkgPUUG7hISoYOZNSuLT/DI27S3nuikpRkcSERER6RNer5fj5Y1sO1BB3qFKWjvcAKQn\nhJE9LoEZY+IJD9ZlI0R6mgraZdyUM5StByr4+9aTZI9LIChAb5mIiIgMXGcb2tnWPYWxsrZrZdGI\nUDtLJqWRPS6BlNhQgxOKDGxqG5cRHmJn6Yw03t58kg92lnDr3GFGRxIRERHpUe1ON3uKqtl2oILC\n4jq8gM1qZsaYeLLHJTBmiAOLWVMYRfqCCtoVWDwtjQ35ZXy4q4T5WclakUhERET8XqfXS1FxHVsP\nVLCnqJoOlweAESkR5IxPZOqoOIID9VFRpK/p/7orEGC3cMucofzxgyLWbjnJd5ZmGh1JRERE5Bup\nqG1l24EzbD9QwdnGDgBiIgJZPC6V7HEJxDmCDU4oMripoF2h2RMS+WhXKZs/L2fRtFSSY0KMjiQi\nIiJyRVraXeQdrmLb/jMcL28EINBuYfaERHLGJTAiNRKzlsYX6RdU0K6QxWzmjvkZvPjWft7aeJx/\nuWOC0ZFEREREvpbb08mBk7Vs23+GvcdqcHu8mICxQxxkj09k8shYAmwWo2OKyFeooF2FScNjGJkS\nwd5jNRSV1DEqzWF0JBEREZHzlFQ2sXZbMRt2l9DY6gIgMTqYnPGJzBqboHPpRfo5FbSrYDKZuHPh\ncJ7+0x7+tuE4P/n2FEyaDiAiIiIGa2hxsqN7afzSqmYAQgKtXDc5hezxCQxJCNNnFhE/oYJ2lTKS\nIpiaGcfuwip2F1UzLTPO6EgiIiIyCLncHvYeO8vW/Wc4cKKWTq8Xi9lE1ogYluYMZUhsCFaLlsYX\n8TcqaN/A7fOGUXCkmrc2HSdrRIy++YmIiEif8Hq9HC9vZNuBCvIOVdLa4QYgPSGM7HEJzBgTT3iw\nndjYMKqrmwxOKyLfhAraNxDvCGb+pGQ+yT/Npr3lXDclxehIIiIiMoCdbWhnW/cUxsraVgAiQu0s\nmZRG9rgEUmJDDU4oRuv0dtLiaqXJ2UyTs5nj7V7amt3YLTZsZjsBFvs5t23YLXbMJg0y9EcqaN/Q\njTlD2HrgDGu3nCR7XAJBAXorRUREpOe0O93sKapm24EKCovr8AI2q5kZY+LJGZfA6CEOLGZ9wB6o\nvF4vHZ4OmpwtNLm6Slezs9l32/en+36LqxUv3qt6DavJgs3SXd7MNmwWW/dtOzaLDbu5676te3/X\nbRt2c1fZs3dvt3c/h637tv0t+krYAAAgAElEQVScx6gEXj21im8oPMTO0pnprPnsBO/vLOa2uRlG\nRxIRERE/1+n1UlRcx9YDFewpqqbD5QFgREoEOeMTmToqjuBAfXzzV55OD82ulvOK1blFq9nZfF4h\nc3W6LvucwdYgwuyhxAfHEWYP7fpjCyHO4aC+sQWnx4mz04XT48LZ6ey673F1b3P69rd7Omh0NuPs\ndNLp7eyxr9lqtvpK3JfFzX7BtojSEDxOzil/5zy2uyx23bafd9tmtg64Eqj/w6/BDdNS2ZB/mo/y\nSlmQlaJla0VEROQbqahtZduBM2w/UMHZxg4AYiICWTwulexxCcQ5gg1OKBfj9Xpp97TT5Gym8YIR\nrhZf6fpiX4u79bLPaTVZCLOHkRgSR6g9lDBbKOH2MELtIYTZQr8sYfZQQm0hWM0X/zh/Lechejo9\ndHic3YXOdU7Jc+LqdHXtu6DwOXF5uvd1unB5nOfc7jq2zd1Oo6eJDo/zqkf7LsVmtnYXt4uXua+O\nDH45Ymi/6CjhF1NB7d2PtfZxCVRBuwYBNgu3zBnGK+8XsnbLCb6zdLTRkURERMRPtLS7yDtcxbb9\nZzhe3ghAoN3C7AmJ5IxLYERqJGYtjd/nXJ3uc4pWi+92o7MJ14kOahrrz5ty6PZ6LvucIbZgwgLC\nSApNOGeUK7SrgNlDCbeHEtpdvgItAYZfEsFithBsDiKYoF55fq/Xi9vrwXVO8QsJt1F5th5nd8lz\nnTvyd05Z7OgugueWxnPLYqurjfrOBpweVw+XQJtvGufyzIXMip7ZY8/9VSpo1yhnfAIf7Spl8+dn\nWDQ1lWSdpCsiIiJfw+3p5MDJWrbtP8PeYzW4PV5MwNihUeSMSyBrZCwBNovRMQeUTm8nbe72C87Z\nOn9a4Zfb29ztl31Om9lGuD2U5LCkC0a1zr0fagsl1BaMxay/03OZTCZsJis2s5UvxoZjHWGEuCN7\n7DW8Xi/uTvcFUzm/WvguNkr4tY/t3u7udPdYzotRQbtGFrOZO+Zn8MKbn/PmxuP8/+6caHQkERER\n6WdKKpvYdqCCHQcraGztOq8oMTqYnPGJzBqboNMkrpLT4+oawXJduFhGk7OFJmfTl+XL1XLZc6pM\nmAi1heAIiCQt7KsjXF9OLRySkICz2USAxd5HX6l8UyaTCZula3pjiK1npwj39mUsVNB6wMSMaEal\nRrLv+FmKSuoYleYwOpKIiIgYrKHFyY7upfFLq5oBCAm0ct3kFLLHJzAkIczwqWz9xblLxH9Rus4/\np6vlvBGvdk/HZZ8z0BJAqD2U9MCo7hGtL4pWWNftc6YVhtiCr+gco9jQMKrbdH056V0qaD3AZDJx\n54LhPPWn3fxtwzF+8u2p+oYrIiIyCLncHvYeO8vW/Wc4cKKWTq8Xi9lE1ogYssclMnF4NFbLwFpx\nDroKVru7gzZ3O+2edlpdbbR72mlzn/unrWv/V7d52mjsaL7s+UJmk5kwWwjRQVFdi2bYQn1F66vT\nCsPsodgttj766kV6lgpaDxmWFM60zDh2FVaxq7CK6aPjjY4kIiIifcDr9XK8vJFt+8+Qd7iK1o6u\n81PSE8LIGZfA9DHxhAf33ylxX1xv66tlqt3dTusXhcpzfsk6v2i1XdGI1leZTWaCrIGEB4QSExjd\nVa6+UrbOvR1kDRxwy6mLXIwKWg+6fd4w8o9U89am40weGTsgf0MmIiIiXc42tLOtewpjZW3X8ukR\noXaWTEojZ1xCnywc5vV6cXa6fIWqrbtU2dugsrbOV6RavyhVnravFLGu/Ve72p0JE0HWQIKsgUQH\nRfluB1mDuv5rCSTwq9u6/wRaAwm2BmEz2zCZTL1+Po+Iv1FB60FxjmAWZCWzfs9pNhSUsWhqqtGR\nREREpAe1O93sKapm24EKCovr8AI2q5kZY+LJGZfA6CEOLOYr/wWty+PqGp1ytZ0zSvV10wHbzyti\nXzz+m1xUONDSVZYcAREEhcT7itNXy9SXRev87QH9YCl4kYFKBa2HrcgZwtYDZ3hn6ylyxiUSHKi3\nWERExN8VldTx5/VH2bqvnA5X13WvhqeEMnVsFJlDwsDiotVdy/6a8vPK09eed9W97UquofVVARY7\nQdYgwuxhxAXFnl+orEEEWgOJjYzA0246bwQr0BJIsK2rXGmqoEj/pfbQw8KD7Sydkc7qz07w/s5i\nbp+XYXQkERG5Ri0tLaxcuZKGhgZcLhcPP/wwsbGxPPnkk5jNZsLDw1m1ahVBQV9e1HX16tU8//zz\npKWlAZCdnc1DDz1k1Jcg31CTs5k3925m5+kDmKxOLGM8RAR00mlyUuZ1U1YHa+uu/PlsZhtB1kBC\nbMHEBEV/OXJluXA64MWmCAZaAq7omlqaNijiv1TQesGiaal8mn+aj3aVsiArmdjYMKMjiYjINViz\nZg1Dhw7lkUceobKykgceeICYmBgee+wxJkyYwM9//nNWr17Nt771rfOOW7ZsGStXrjQotXxTTo+T\nz2sOsauigEO1RXR6O7FEgsVkIdgaRJA1mEBr1OULleX8c64CrQFYzfroJSKXpu8SvSDAZuHWOcP4\nn/cLeXvLSVZmxBodSUREroHD4aCoqAiAxsZGHA4Hv/3tbwkN7VoEIioqivr6eiMjyjXq9HZSVHeM\nXRUF7K3eT4fHCYDDEkfliSiyU7L493+Yq1EpEel1Kmi9JGd8Ih/tKmXr/jMUn2kk2KoTaUVE/NXy\n5ctZvXo1ixYtorGxkd/97ne+ctba2sratWt5/vnnLzguLy+P7373u7jdblauXMmYMWP6Orpcgtfr\n5XRzOXkV+eyp3EuDs6t8RQc6WJAym6zYSTz/5+OYW13ccof+7kSkb6ig9RKz2cSdCzL41Ruf88q6\nQ/zzzWONjiQiIt/Q2rVrSUpK4uWXX6awsJDc3FxWr15Na2srDz30EA8++CAZGeefczxx4kSioqKY\nP38+BQUFrFy5knfeeeeSr+NwBGO1Xv78osvxp6n1RmStajnLluI8NhfnUdZYAUCIPZhFGXOYkz6D\nUTHDMJlMvL/9FGcbO7hpzjBGDosxLO83pay9Q1l7jz/l7c2sKmi9aPywaDLTItl9uJLDk5IYne4w\nOpKIiHwD+fn5zJ49G4DMzEyqqqpwOp388z//MytWrOC222674JiMjAxfacvKyqK2thaPx4PF8vUF\nrK6u9Zqz+tPiEH2ZtcXVSn7V5+yqyOd4wykArGYrWXETmB6fxZjoUb7zw2pqmnG5O3nto0JsVjPz\nJyZSXd2k97aXKGvv8Kes4F95eyLrpQqeClovMplM3LlgOP/5x928seEYP3lgKmZdM0RExO+kp6ez\nb98+Fi9eTFlZGSEhIbz88stMnz6dO++886LH/OEPfyAxMZEVK1Zw5MgRoqKiLlnOpOe5PC72nz3M\nrooCDp4txOP1YMLESMdwpsVnkRU3jiBr0EWP3fJ5ObWNHdwwLZXI0IA+Ti4ig5kKWi8bmhjO3EnJ\nfLa3jF2Hq5gxJt7oSCIicpXuvvtucnNzue+++3C73fzsZz/j0UcfJSUlhe3btwMwY8YMfvCDH/DQ\nQw/x0ksvceONN/Loo4/y2muv4Xa7efrppw3+KgaHTm8nx+pPkFdRQEHVfto97QAkhyYyLT6LqfGT\ncARGXvI5XO5O3t1ejN1qZunM9L6ILSLio4LWB+5fNpqtn5fz1qbjTB4Zi82qi0OKiPiTkJCQCxYB\n2bJly0Uf+9JLLwGQkJDAq6++2uvZpEtZ8xl2VRSwq7KA+o4GABwBkcxJnsm0hCySQxOv+Lk+21dO\nXVMHi6enEhFi763IIiIXpYLWBxKiQ1gwOZn1u0+zsaCMRdNSjY4kIiLi9+ra69lduZe8inzKW7oW\n+wiyBpKdOJ3pCVlkRA7FbLq6X4q63B7e29E9ejZDo2ci0veuqKA988wz7Nu3D5PJRG5uLhMmTPDt\nO3PmDD/60Y9wuVyMGTOGJ598stfC+rMbs4ewdf8Z3tl2ipzxCQQH2oyOJCIi4ndaXW3srd5PXkU+\nx+pP4sWL1WRhYuw4psdnMTY6E5vlm/+M/WzfGeqaOlgyPY1wjZ6JiAEuW9Dy8vIoLi7m9ddf5/jx\n4+Tm5vL666/79j/33HM8+OCDLFq0iCeeeILy8nKSkpJ6NbQ/Cgu2s2xmOm9tOsF7O0q4Y37G5Q8S\nERERXJ1uDp4tZFdFAQfOHsbd6QZgeORQpsdPJituPMG24Gt/HbeHddtPYbeZWTIj7ZqfT0Tkm7hs\nQdu+fTvXX3890LVkcENDA83NzYSGhtLZ2cmePXv45S9/CcDjjz/eu2n93KKpqXyaX8bHu0tZODmZ\nqPBAoyOJiIj0S53eTk40FJNXkU9+1ee0udsASAiJZ0b8ZKbETyI6qGcvX7Nxbzn1zU6WztDomYgY\n57IFraamhrFjv7zIclRUFNXV1YSGhlJbW0tISAjPPvssBw8eZOrUqTzyyCO9Gtif2W0Wbp0zjP9+\n7zBvbz7Jg8tHGx1JRESkXznTUkleRT67Kgqo66gHIMIeTnbqNKYlTCYlNBFTL1yyxuny8N72YgJs\nFo2eiYihrnqREK/Xe97tyspKvv3tb5OcnMw//dM/sXHjRubPn/+1xzscwVit134dGH+60jh8mfem\nBaF8WlDG1gNnuHtxJkMSww1OdiF/em+Vtff4U15l7R3+lFX8W31HA7sr97KrooDTzeUABFoCmJkw\nlWkJWYx0ZFz1Yh9Xa+PechpanCybmU5YsEbPRMQ4ly1ocXFx1NTU+O5XVVURGxsLgMPhICkpibS0\nrt80zZo1i6NHj16yoNXVtV5jZP+60jhcmPeW2UP51Rv7+P3qz/nXuyYamOxC/vTeKmvv8ae8yto7\neiqrSp58nTZ3OxtPHuCTo9s4UnccL17MJjPjY0YzLX4y42PGYL+GxT6uRoera+XGALtGz0TEeJct\naDk5Obz44ovcc889HDx4kLi4OEJDQ7sOtlpJTU3l1KlTDBkyhIMHD7J8+fJeD+3vxg+LYnS6g/0n\nznLoVC1jhkQZHUlERKTXuTvdHK49Ql5FPvtrDuHqXuxjWEQ60+InMzluAqH2kD7PtamgjMYWJ8tn\npRMapFWWRcRYly1okydPZuzYsdxzzz2YTCYef/xxVq9eTVhYGIsWLSI3N5fHHnsMr9fLyJEjWbhw\nYV/k9msmk4k7F2Tw5Cu7eWPDcf7jOw7MvTCfXkRExGher5eTjcXsqihgT9U+WlxdM2nig2OZP2wm\nY8LGEBMUbVi+DpeH93aWEGi3sHi6Rs9ExHhXdA7av/3bv513PzMz03c7PT2dv/71rz2bahAYkhDO\njDHx7DxUSd7hSmaOSTA6koiISI+pbKliV2UBuyoKqGmvBSDMHsqC1NlMi88iLSyFuLhww6f9bsjv\nGj1bka3RMxHpH656kRDpObfNHcbuwipWbzrBlJFx2Ky9ewK0iIhIb2p0NrGnch95FfmUNJ0GwG6x\nMz1hMtPisxjlGI7FfO0LhfWUDqeH93cWExRg4YZpGj0Tkf5BBc1AsZFBXDclhY92lbIh/zQ3aGqF\niIj4mXZ3B5/XHCSvIp/C2qO+xT7GRI9ievxkJsSOJcDSP1dF/LTgNE2tLm7MHqLRMxHpN1TQDLYi\newibPz/DO9tOkTMhkZBA/YAQEZH+zdPpobDuKHkV+XxefRBnpwuA9PBUpsdPZkr8RMLsoQanvLR2\np5v3d5QQFGDlhumpRscREfFRQTNYaJCNFbPSeWPjcd7bXsydC4YbHUlEROQCXq+X4qZS8ioKyK/c\nR5OrGYCYoGimx2cxLSGLuOBYg1NeuU/zy2huc3FTzhD9clRE+hUVtH7guikprN9zmo93n2bh5BSi\nIwKNjiQiIgJAVWsNuyoL2F1RQFVb13VRQ20hzEvJZlr8ZIaEp2Lys5WI2zrcfLCze/RsmkbPRKR/\nUUHrB+w2C7fNHcbL6w7z9uYTfHfFGKMjiYjIINbkbGZP1T52VxRwsrEEAJvZxtT4SUyLz2J01Mh+\ntdjH1fo0/zTNbS5umT2UYI2eiUg/o4LWT8wam8CHeaVsO1DBommppMWHGR1JREQGEafHyefVB9lV\nWcCh2iN0ejsxYWJ01EimxWcxMXYsgVb/n+HxxehZcICV66dq9ExE+h8VtH7CbDZx14IMfvm3fby5\n8Tg/unuS0ZFERGSA6/R2UlR3jF0VBeyt3k+HxwlAWlgy0+KzmBI/iYiAcINT9qxP9pympd3NLXOG\nEhyoj0Ei0v/oO1M/MnZoFKPTHRw4WcvBU7WMHRJldCQRERlgvF4vpc1l7KooYHflXhqdXReKjg50\nsCBlNtMSskgIiTc4Ze9o63DzYV4JIYFWFmn0TET6KRW0fsRkMnHXguE88cou3thwjNHfmYbZz068\nFhGR/qmmrZbdlQXkVRRQ2VoFQIg1mNnJM5keP5lhEel+t9jH1Vq/u5SWdje3zh1GUIA+AolI/6Tv\nTv1MekIYM8fGs+NgJTsPVTJrbILRkURExE81u1ooqPqcvIoCTjScAsBqtpIVN4Hp8VmMiR6F1Tw4\nPgq0trv5MK+UkEAr109JMTqOiMjXGhzflf3MbXOGsbuwitWbTjB1VCw2q/+ulCUiIn3L6XaSX/U5\neRX5HDpbhMfrwYSJkY7hTI/PYlLcOIKsQUbH7HPrd5fS2uHm9nkaPROR/k3fofqhmMggrpuSwod5\npXyyp4wlM9KMjiQiIn5gQ+kW1m3+iDZXOwDJoYlMT5jMlLiJOAIjDU5nnNZ2Fx/uKiU0yMbCyRo9\nE5H+TQWtn1o+awib951h3fZTzJmYSIiu0yIiIpdR3lxBmD2EOUmzmBafRVKopskDfLz7NG0dbu6Y\nn6HRMxHp9/Rdqp8KDbKxPDudNzYcZ932Yu5aMNzoSCIi0s99a/QdxMaGUV3dZHSUfqO13cVHvtGz\nZKPjiIhclgpaP3b9lBQ+3XOa9btPs3ByMjERg++cARGR/qClpYWVK1fS0NCAy+Xi4YcfJjY2lp/9\n7GcAjBo1iieeeOK8Y1wuF4899hjl5eVYLBaeffZZUlO1tHtf+2hXKW0dbu6cn0GgXR97RKT/Mxsd\nQL6ezWrh1rnDcHs6WfPZSaPjiIgMWmvWrGHo0KG8+uqrPP/88zz99NM8/fTT5Obm8tprr9Hc3Mym\nTZvOO+bdd98lPDycv/71r3z/+99n1apVBqUfvFraXXy8u5SwYJ17JiL+QwWtn5s5NoHUuFB2HKyg\npFJTVkREjOBwOKivrwegsbGRyMhIysrKmDBhAgALFixg+/bt5x2zfft2Fi1aBEB2djb5+fl9G1r4\nMK+Utg4PS2ekE2DXisgi4h9U0Po5s8nEnQsy8AJvbDxudBwRkUFp+fLllJeXs2jRIu677z7+/d//\nnfDwcN/+6OhoqqurzzumpqaGqKgoAMxmMyaTCafT2ae5B7PmNhfrd5cSHmxjQZbOPRMR/6HJ2H5g\n3NBoxg5xcPBkLQdOnmXc0GijI4mIDCpr164lKSmJl19+mcLCQh5++GHCwsJ8+71e72Wf40oe43AE\nY+2Ba1/GxoZd/kH9RG9lff+9Q7Q7PXxrSSYpyT13iQG9t71DWXuHP2UF/8rbm1lV0PzEHfOHc+iV\nXbyx4ThjhkRhNpmMjiQiMmjk5+cze/ZsADIzM+no6MDtdvv2V1ZWEhcXd94xcXFxVFdXk5mZicvl\nwuv1YrfbL/k6dXWt15zVn1Zx7K2szW0u/r75BOEhdqaOiOmx19B72zuUtXf4U1bwr7w9kfVSBU9T\nHP1EekIYM8cmUFrVzI6DFUbHEREZVNLT09m3bx8AZWVlhISEkJGRwe7duwH46KOPmDNnznnH5OTk\n8MEHHwCwYcMGZsyY0behB7EP80rocHpYNjOdAJvOPRMR/6IRND9y69yh7CqsZM1nJ5iWGYetB6bB\niIjI5d19993k5uZy33334Xa7+dnPfkZsbCw//elP6ezsZOLEiWRnZwPw0EMP8dJLL7Fs2TK2bdvG\nvffei91u57nnnjP4qxgcmlqdrN9zmogQO/MnJRkdR0Tkqqmg+ZGYiCCun5LKB3klfLKnjCUz0oyO\nJCIyKISEhPD8889fsP1///d/L9j20ksvAfiufSZ964Pu0bPb5gzDrtEzEfFDmuLoZ5ZnpxMSaOXd\nbadobnMZHUdERKTfaGx18umeMiJC7czT6JmI+CkVND8TEmhj+awhtHa4Wbf9lNFxRERE+o0PdpbQ\n4fKwfGa6Rs9ExG/5XUGr72jgNzv/xI4zu3F6BucI0nVTkokOD+CTPaepqW8zOo6IiIjhGlucfJp/\nGkdYgEbPRMSv+WVB+6x4J68e/hu5W5/ijSNrOdNSaXSsPmWzWrhtbgZuj5c1m08YHUdERMRw7+8s\nxunqZNnMdC2iJSJ+ze8K2pDwNF5Y9gSL0xdiM1vZeHorT+1cxS/3/IadZ/YMmlG1GWPjSYsLZfvB\nSoor/OOaESIiIr2hocXJhvwyHGEBzJ2o0TMR8W9+V9AA4kJjuCljCU9l5/K9cfeT6RjB8YZT/Onw\n6/x461O8efTvVLRUGR2zV5lNJu5cOByAv204htfrNTiRiIiIMd7fUYzT3cmKWenYrH750UZExMev\nl9m3mC1kxY0nK248NW1n2Vqex/byXWwo3cKG0i0MjxzK7KSZTIobj83s11/qRY0dEsW4oVEcOFnL\nwZO1jBsWbXQkERGRPtXQ3MHGgjKiwgOYPUGjZyLi/wZMa4kJiubmjKUsH7qIz2sOsaVsB0V1xzhW\nf5KQo2uZmTCVnKTpxIfEGR21R90xP4ODJ2t5Y+NxxgyJwmw2GR1JRESkz7y3owSnu5Pls4Zo9ExE\nBoQBU9C+YDVbmRw3gclxE6hqrWZreR47zuzmk9LP+KT0M0ZEDmN28kwmxo4bEKNqafFhzBqXwLYD\nFWw/WEHO+ESjI4mIiPSJ+uYONu4tIzo8gDkT9PNPRAYG/28olxAXHMutw5ezYthi9lUfYGvZTo7U\nH+do/QlCbSHMTOwaVYsLjjU66jW5dc4w8g5XsWbzCaZlxunaLyIiMii8t70Yl7uT5dlDsFo0eiYi\nA8OALmhfsJmtTI2fxNT4SVS2VrO1bCc7KnazvmQT60s2MdIxnNlJM5gYOxarH46qRUcEsmhqCu/v\nLOGTPadZOjPd6EgiIiK9qq6pg417y4kOD2S2Zo+IyADif23kGsUHx3LbiBXcOGwxe6sPsKV8B0fq\njnGk7hhhttDuUbUZxAb714Iby2al89m+ct7dXsyciUmEBtmMjiQiItJr3ttRjNvTyY05Gj0TkYFl\n0BW0L9gsNqYlZDEtIYuKlkq2luex88wePi7ZyMclG8l0jGB28kwmxIzBYu7/UwZDAm2syB7C658e\n491tp7jnuhFGRxIREekVdU0dbNpbTkxEINnjEoyOIyLSowZtQTtXQkg8t4+4kZuGLaGgej9bynZQ\nWHeUwrqjhNlDmZU4jZykGcQERRkd9ZIWTk7hkz2n+TT/NNdNSSE2MsjoSCIiIj1u3fZTuD2drNC5\nZyIyAOm72jlsFhvTEybzoyn/zI+n/4j5KTm4Oz18VLyBn23/Ob/e+//YW30AT6fH6KgXZbOauW3u\nMNweL2s+O2F0HBERkR5X29jOZ/vKiY3U6JmIDEwaQfsaSaEJ3DnyZm7OWEZB1edsKd/B4dojHK49\nQoQ9jFlJ08lOnE50kMPoqOeZPiaeD/NK2XGokhumpzIkIdzoSCIiIj1m3fZi3B6vRs9EZMDSd7bL\nsFtszEicwiNTHiZ3+r8yLyUbZ6eLD059wuPbn+O/9r3MvuqD/WZUzWwyceeCDADe2HAcr9drcCIR\nEZGecbaha/QsLjJIo2ciMmBpBO0qJIcmctfIW7glYxl7KvexpXwnh84WcehsEZEBEd3nqk3HERhp\naM4xQ6IYNyyKAydqOXCylvHD/GtFShERkYtZt/0Unk4vN+YMwWLW75hFZGBSQfsG7BY7s5KmMStp\nGqebytlavpO8inzeP7WeD059wtjoTGYnz2BsdCZmkzE/QO6cP5yDJ/J4Y8Mxxg6Jwmw2GZJDRESk\nJ9Q0tLH58zPEO4KYOTbe6DgiIr1GBe0apYQlcfeoW7ll+HL2VO5lS9lODpw9zIGzh4kMiCA7aTo3\nhiygr9/q1LhQsscnsHV/BdsOVDB7gi7iKSIi/mvd9mKNnonIoKCC1kMCLHayk6aTnTSd0qYytpTt\nYFdlAe+d/Jj3T61nXPRoZifNYEz0qD4bVbt1zjDyDlexZvMJpo+Ow27r/9dzExER+aqa+ja2fH6G\n+KhgZozR6JmIDGwqaL0gNSyZezNv59bhK9hTuZcdVbvYX3OI/TWHcAREkpM0g1lJU4kMiOjVHFHh\ngVw/NYX3d5Swfs9pls1M79XXExER6Q3vbu869+ymbI2eicjAp4LWiwKtAeQkz+CWSdez5/hhtpTv\nYFflXt49+SHvnfqY8dGjyUmeyeioEb02qrZ8Zjqf7S1n3fZTzJmQSFiwvVdeR0REpDdU17exdX8F\nCRo9E5FBQgWtj6SFp/AP4Xdw6/AV7K4sYEvZTvbVHGRfzUGiAx1kJ81gVuI0IgLCevR1gwNt3Jgz\nlNc+Ocq724q59/oRPfr8IiIivemdbd2jZzlDtOCViAwKV1TQnnnmGfbt24fJZCI3N5cJEyb49i1c\nuJCEhAQslq7zm37xi18QH6/fcH2dIGsgc5JnMTtpJiVNp9lStoPdlXt558QHrDv5ERNixjA7aSaj\noob32Kjagqxk1u8u5dP801w3NYW4yKAeeV4REZHeVFXXyrb9FSRGBzN9tD5biMjgcNmClpeXR3Fx\nMa+//jrHjx8nNzeX119//f/f3p2HR1nf+/9/ziSZrJNlkpkkZCcJWyIQ9n1RQETciwdPbeu39rQV\nPb1Ov7aXltqLc/y51H7BS2utbfFqT+vPU63IVxEXUETKmrBDCJCFANlIJnvClm2+fySMBEiCkGRm\nktfjurjIPXPPzWs+Iu3JBDcAACAASURBVPe8531/7k+nfVavXk1gYGCfhRyIDAYDCcFxJATHcX/q\nYnaf2c+20kwO2LM5YM8mws/C9CGTmTJkAsGmm+uq+XgbuX/2UP60Lof/+88T/OjutF56FyIig8N7\n773HunXrnNsHDx5kzJgxzu2Kigruu+8+fvzjHzsfe+211/joo4+cX1refffdLFmypP9CDwAf7ThJ\nm8PB3dOT1D0TkUGjxwJt586dzJs3D4Dk5GTq6upobGwkKCioz8MNFv7e/syKncbMmKmcrD/NtpJM\n9lYc5MMTn7K+cCOjrWnMGDKZYWHJN9xVmzQykg1ZRWTmlLNgYhxJ0cG9/C5ERAauJUuWOIurrKws\nPv30U1asWOF8/gc/+AH33HPPVa/77ne/y8MPP9xvOQeS8ppz7MwuZ0hEIBNH2FwdR0Sk3/T4ab+y\nspKwsDDntsViwW63d9pnxYoVPPTQQ6xcuRKHw9H7KQcJg8FAUkgC3xn1IC9Mf4Ylw+4hMsDK/opD\nvHZgNc/u+j98fuorGpoav/GxjQYDD85NAeC9zfn67yQicoNef/11li1b5tzesWMHiYmJREdrvcne\ntH77pe6Z5p6JyODyjW8ScuUH+5/85CfMnDmTkJAQHn/8cTZs2MDChQu7fH1YWADe3je/HpfV2rs3\n0+hr3zyvmYQhC/nW2NvJrTrB5wVb2Vm0jw8KPuGjwg1MjhnLvOSZpNmGYTBc34nLajWz+UApe46W\nc7rqPBO6uJ7fk8ZWWfuOJ+VV1r7hSVn7y6FDh4iOjsZqtTof+9vf/sby5cuvuf9nn33Gpk2bMJlM\nPPPMM8TFxfVXVI9WXn2OHUfOEBMRyAR1z0RkkOmxQLPZbFRWVjq3KyoqOp2Y7r33XufPs2bNIjc3\nt9sCrabm3I1mdbJazdjtDTd9nP5ys3kt2PiXoQ+wOO4Oss7sY1vJLnYU7WVH0V5sARHtc9WiJhBk\n6nke4N1TE9h7tJw3PzxMnMX/qm8lPWlslbXveFJeZe0bvZV1oBV5a9as4b777nNul5eXc+7cOeLj\n46/ad/bs2UyZMoWJEyfy8ccf89xzz/HHP/6x2+MPxi8xr5X1rc9zcTjg4UUjibS51yX5nj627kpZ\n+4YnZQXPytuXWXss0KZPn85rr73G0qVLOXLkCDabzTn/rKGhgf/4j//gjTfewGQysXv3bm6//fY+\nCzvYBfoEMDduBnNip1NQd5JtJZnstx/i/+Z/zEcFnzHWdgszhkwmJXRol121WFsQ02+JZtvhMrZn\nlzFz9JB+fhciIp4rMzOTZ555xrm9ZcsWpkyZcs19r7zj8cqVK3s8/mD7EvNaWcuqzvLVvmJirYGk\nRrvXe/H0sXVXyto3PCkreFbe3sjaXYHXY4E2btw40tLSWLp0KQaDgRUrVrB27VrMZjPz589n1qxZ\n/Mu//Au+vr6MGjWq2+6Z9A6DwUBKaBIpoUl8q/kussr2sq00kz3lB9hTfoDIABszhkxicvQEAn0C\nrnr9vTOTyDxazgdbC5k0MhJfn5v/tlZEZKArLy8nMDAQk8nkfOzw4cPMnTv3mvs/99xzLFy4kAkT\nJpCVlUVqqtahvB4f7TiJw0H7nRuv8xJ+EZGB5LrmoP3sZz/rtD1ixAjnz9/73vf43ve+17up5LoF\n+QRya/ws5sbNJL/2RPut+isO837+ej488RkZ1tHMiJlMckiis6tmCfZjwcQ4Pt55ii/2FHHn1ETX\nvgkREQ9gt9uxWCxXPRYeHt5p+7XXXuPZZ59lyZIlrFixAm9vbwwGA88991x/R/Y4ZVVnycwpJ9Ya\nxLjh1p5fICIyAH3jm4SIezIYDKSGJZMalkxj6ll2ndnD9tJMdpfvY3f5PqICI5kxZDKTo8YR4BPA\nHZMT2HKglE92nWLWmCGYA0w9/yEiIoNYeno6b775ZqfH/vCHP3TatlqtPPvsswAMHz6cd955p9/y\nDQTrtrd3z+6Zoe6ZiAxeKtAGoCBTIPPiZ3Nb3CzyagvYVtK+APaavHV8WPAJ42xjmBEzmcVTE3jn\ny3w+2nGSf503zNWxRURkECutPEtWTjnxtiDGDYtwdRwREZdRgTaAGQwGhoWlMCwshYamRnaV7WFb\naSaZZ/aSeWYv0YGRhCRGsvlgC/PGx2ILu3q+moiISH9Yt70QB+3ds+tdPkZEZCBSgTZImE1BzE+Y\nw23xs8itKWBbaSYH7dm02crxCTfy26xCHp2ykIiINFdHFRGRQabE3sjuoxXERwYxNlXdMxEZ3FSg\nDTJGg5ERllRGWFKpb2pgR+luPsndSo1PASv3vo45O4hgHzMhvsGEmoIJ8e34ddnPZp8gvIy686PI\nQNHmaONc83nqmxpobG6koamRsItBJJmS1cmQfrFu+0l1z0REOqhAG8SCTWYWJt5KAmNZ9ckXhCWU\nE2S+iP1cFSWNZV2+zoCBYFPQNYu39p9DCPUNJtAnAKPB2I/vSEQuaWptpqGpgYaOgquh6Wyn7cam\ns9R3bDc2ncWB46pj/OeUp7AGhF/j6CK9p9jeyJ5jFSREmRmbou6ZiIgKNGFkooV063AOHY7g8Ucn\nk2gN5ELLBeou1lPXVE/txXrnz3UXv/5Vdrac0w0lXR7XaDBeo3gLvqo7F+Dtr29MRXrQ5mjjbPO5\n9uKquZH6pkuFViMNHduNHY81NDdysbWpx2P6eflhNgViDYnAbArC7BPY/rvJzLDoeCKMlh6PIXKz\n1m3T3DMRkcupQBMAvjUnmcMnqvjNW3uYcUs08ybEEhlmIzLQ1uVrHA4H51vOtxdwlxdvl/1ce7Ge\nooYSTtaf7vI43kbvLou3yx/38/LVyVsGlKbWJmdB1dB02a9rbHfV5bqc0WDE7BOI1b+94AryCcJs\nCiTYZCbosgIs2GQmyCcQHy+fLo9ltZqx2xt6+y2LdFJYWsee43aSos2MSVa3VkQEVKBJh1hrEP/r\njpF8uL2QL/YWs2lvMWNTI1gwMY5hcaHXLIwMBgMBPgEE+AQwhKguj33pm/+rircrOnKFdae6/QBq\n8jJ1Kt6iQiMwtfp1KuRCfYMxeWlNN3GNy7tclxdajU3tHa6mYxeoaqyloamR+uZGmq6zyxVsCrqi\ny2Xu6HR13vb39tNlxeJR3vn8OKDumYjI5VSgidOM0dHcNSeFT7cV8PnuIvbnVbI/r5KESDMLJsYx\ncaQNb69v/uHPaDA6P0zGMqTL/docbTQ0NV59aeUVhV3F+cr2F5Rf+zj+3n5dX1rZsR3sG4yPUX/9\npWeXulz1HZcWdtfpamy+3i5XELaOLld7kdX+e5ApiODLt3vocol4stPlDew4VMbQIcHcMlTdMxGR\nS/QJVTrx9jIyZVQUk0dGkl9Sx8bdRezLtbN6fQ7/+Cqf28bFMicjhiD/3v/QaDQYnYVUd1raWmho\nasQQ0MLJ8jOdunCXF3JnzlV0e5xAn4CuL628VMiZzLpj5QDTVZer8+WEHXO8rrPL5e/th9knCGtA\nBMEdhdalIuvyAiwpOopzda3qFIjQfudGUPdMRORKKtDkmgwGA6mxoaTGhmKvPc+mvcX882Apa/95\ngvU7TjItPYr5E+OIDg/s92zeRm/C/EKxhpsJbev6jl/Nrc3UNTVcc17cpe3qC7WUnj3T5TEMGAgy\nBXY5L659OwSzKVCXlrlQU2vT1zfN6Ci4nDfNuIkuV6R/RHuxdWXBdUXX63q7sUG+gZw3aF6XyOny\nBvbl2hkeH0Z6km5GIyJyORVo0iNrqD9Lb0vlnhlJbD1Uxhd7ivjqQClfHSjllqHhLJgYx6jEMLf7\nBtTHy4cIfwsR/t2f/C+0XKS+q7lxTZe6cXaKGku7PIbRYCTYZO7y0sp4Lxu1DedwOBw4cNDW8fvX\n223Onx0OB204cDjarti+9HzbNR5z0Ma1jnHZ9mXH62kfvxIfzp27eNU+V+b8+r20dfHersh6jfd2\nVRZH21W5rtzn8vdyoe0iF1su9vj3wd/bD7MpCFtAxNeXE15WZJl9Oi4vNAXhrzuLivSpD7cVAvCv\nt4/Q/2siIldQgSbXzd/XmwUT45g3Ppb9eXY27i7i8IkqDp+oIsYayPwJcUxNi8TH27MuCfTz9sXP\n24otwNrlPg6HgwutF77uwHVx58qSxlJONRT1Y3rPZsCAwWDA2PG7wfm7sfPjzp+Nl73GiMFowBIQ\ngp/R/6rO1uXb36TLJSJ969SZBvbnVZIcE0zGcCuVlY2ujiQi4lb0iUW+MaPRwPjhNsYPt1FYVs/n\nu4vYfayC//70GO9vKWBuRgxzx8USEjhw7qZoMBjw9/bH39ufqMDILvdzOBycbTl31by4Fq8mLpxv\nvqIIuaLwuGah0rFPp8Lkytcbr3G8r4sZ41XHM3Y6hvGyfS89ZgkLpK72/FV/tuHK19HxfJfvq73Y\nujrD17/fLN0OXsSzXOqe3TtjqLpnIiLXoAJNbkpSdDA/vDuNb81J5st9JWw5UMK67Sf5ZNcpJo+K\nZP6EOOIjza6O2W8MBgNBPoEE+QQSExTtfNzTighruBl7m+fkFRHPUFhWz4H8SlJiQxiVGObqOCIi\nbkkFmvQKS7Af35qTzF3TEtmRXcbGPcVsP3yG7YfPMDIhjPkT4xidHI5R35aKiAxa65zdM925UUSk\nKyrQpFf5mryYOy6W2RkxHC6oYuPuIo6equHoqRoiLQHMnxDL9PRofE2eNU9NRERuTmFZPQcLqhgW\nG8LIBHXPRES6ogJN+oTRYGBMSgRjUiIoqmjk891F7Mo5w/+/MZe1W04we+wQbhsfiyXYz9VRRUSk\nH1yae6Z1z0REuqcCTfpcnC2I7985kgfmJLN5XzGb95fwaeZpNmQVMWGElQUT4xk6pPvFqUVExHMV\nlNZxqKCKYXGhjFD3TESkWyrQpN+EBJq4d+ZQ7pyawK4j5WzcU0TW0QqyjlaQEhPCgolxZAyLwMuo\nBZ9FRAaSDzX3TETkuqlAk37n4+3FzDFDmDE6mpxTNXy+u4hDBVXkl9QRHuzHvAmxzBw9hAA//fUU\nEfF0+SV1ZJ+oZkS8umciItdDn4DFZQwGA2mJFtISLZRVneXzPcXsOFzGu1/m8+G2QmaMjmbehDhs\nof6ujioiIjfo8rlnIiLSMxVo4haiwwP57u3DuX/WULYcKGHT3mK+2FPMpr3FZKRaWTAxjtTYEF0a\nIyLiQfKL6zhSWM3IhDCGx6t7JiJyPVSgiVsJ8vfhzqmJ3D4pnt3HKti4u4h9uXb25dpJiDKzYGIc\nE0fY8PbSPDUREXf34bYTgLpnIiLfhAo0cUveXkampkUxZVQkecV1bNxdxP5cO6s/yuG9zfncNj6W\nB+YNd3VMERHpQl5xLUdO1jAqMYxhcaGujiMi4jFUoIlbMxgMDIsLZVhcKBW15/liTxFbD5Xx/pYT\nfLTjFNPSo5g/IZbo8EBXRxWRAey9995j3bp1zu3s7GzS09M5d+4cAQEBADz11FOkp6c792lububp\np5+mtLQULy8vXnzxReLi4vo9u6t8sFVzz0REboQKNPEYtlB//nXeMO6dMZSth0rZvL+Erzp+jU4O\nZ/7EOEYlhGmemoj0uiVLlrBkyRIAsrKy+PTTT8nPz+fFF19k2LBh13zN+vXrCQ4OZtWqVWzbto1V\nq1bxyiuv9Gdsl8ktquXoqRrSEsNIjVX3TETkm9BEHvE4AX7e3D4pnj/9Yh7L7k0nJSaEQwVVrHrn\nACv+nMXWg6U0t7S6OqaIDFCvv/46y5Yt63G/nTt3Mn/+fACmTZvGvn37+jqa2/hga8fcs5lDXZxE\nRMTzqIMmHsvLy8iEETYmjLBxorSejbtPs+eYnb98eoz3txQwJyOGueNiCQk0uTqqiAwQhw4dIjo6\nGqvVCsBvf/tbampqSE5OZvny5fj5+Tn3raysxGKxAGA0GjEYDDQ1NWEyDex/k46fruHY6VrSkyyk\nxIS4Oo6IiMdRgSYDwtAhwfz4nnSq515g095ithwoZd32k3yy6xRTRkWxYGIcsbYgV8cUEQ+3Zs0a\n7rvvPgC++93vMnz4cOLj41mxYgVvv/02jz76aJevdTgcPR4/LCwAb2+vm85ptZpv+hg36uX3DgLw\nyF1p15XDlVlvhCflVda+oax9x5Py9mVWFWgyoFiC/VgyN4W7piey/fAZvthTxLbDZWw7XMbIhDAW\nTIzjluRwjJqnJiI3IDMzk2eeeQbAefkiwK233sonn3zSaV+bzYbdbmfEiBE0NzfjcDh67J7V1Jy7\n6YxWqxm7veGmj3Mjjp2qIbugiluGhmMJ8Okxhyuz3ghPyqusfUNZ+44n5e2NrN0VeJqDJgOSn8mb\n28bH8vwPp/CTB0YzIj6Uo6dqeHXNIZ5ZncnmfcVcbNI8NRG5fuXl5QQGBmIymXA4HDzyyCPU19cD\n7YVbampqp/2nT5/OZ599BsDmzZuZPHlyv2fuTw6Hgw+26c6NIiI3Sx00GdCMBgNjUyMYmxrB6fIG\nPt9dRObRct7amMvaf55g9tgYbhsfS5jZ19VRRcTN2e1255wyg8HAgw8+yCOPPIK/vz+RkZH8+7//\nOwCPPfYYb7zxBosWLWLHjh089NBDmEwmfv3rX7syfp87dqqG3KJaRieHM3RIsKvjiIh4LBVoMmjE\nR5p5dPEovjUnmS/3lbB5fwmf7DrFhqzTTBxhY/7EOJKi9aFCRK4tPT2dN99807m9aNEiFi1adNV+\nb7zxBoBz7bPBQN0zEZHeowJNBp2QIF/umzWUO6cmsCunnM93F7Erp5xdOeWkxoawYGIcGalWjEbN\nUxMRuR45p2rIK65jTHK4vugSEblJKtBk0DL5eDFrzBBmjo7myMlqNu4uIvtENXnFdUSE+DFvQhwz\nR0fj76v/TUREuuJwOPhwa0f3bKa6ZyIiN0ufPGXQMxgMpCeFk54UTmnlWT7fU8SO7DO8symPD7ae\nYNaYIcwbH0tEqL+ro4qIuJ0jJ6vJL6ljbEoEiVHqnomI3CwVaCKXGRIRyPcWjuD+WUP56kApX+4r\nZuPuIj7fU8S4YVYWTIwjJSYEg27TLyLSuXumuWciIr1CBZrINZgDTNw1LZE7JseTdbScjbuL2Hvc\nzt7jdpKizcyfGMeE4Ta8vbRShYgMXkcKqykorScjNYKEKM9ZYFZExJ2pQBPphreXkWnp0UxNiyK3\nqJaNu4s4kFfJn9bl8J65gNvGxzJ77BAC/XxcHVVEpF/pzo0iIn1DBZrIdTAYDAyPD2N4fBjlNef4\nYk8x2w6VsearAtZtL2T6LdHMnxBHlCXA1VFFRPrF4RPVnCitZ9wwK/GR6p6JiPQWFWgi31BkWADf\nnj+M+2Ym8c+DZWzaW8TmfSVs3lfCmORwFkyMY0RCmOapiciA5XA4+HDbCUDdMxGR3qYCTeQGBfj5\nsHByPPMnxrL3uJ3PdxdxsKCKgwVVxFqDWDAxjsmjIvHx1jw1ERlYDhVUUVjWwPjhVuJsQa6OIyIy\noKhAE7lJXkYjk0ZGMmlkJAUldXy+p4g9x+z8+ZOjrNlSwK0ZMTwwb7irY4qI9Ir27lnH3LPp6p6J\niPQ2FWgivSg5JoTkmBCq5lxg075ithwo5YNthazfeYqRCWFkpEYwNjWC0CBfV0cVEbkhB/OrOHmm\ngQkjbMSqeyYi0utUoIn0gfAQPx6cm8Ld0xPZfvgM27PPcPhEFYdPVPG3DccZOiSYjNQIMlKtRIcH\naL6aiHiES90zA3DP9ERXxxERGZBUoIn0IT+TN7eNj2XpwpHk5FVwIK+S/Xl2covqOFFaz/tbThBp\nCSAjNYJxqVaGDgnGaFSxJiLu6UB+JafKG5g00kaMVd0zEZG+cF0F2gsvvMDBgwcxGAwsX76c0aNH\nX7XPqlWrOHDgAG+99VavhxQZCKyh/syfGMf8iXE0nm/mUEEl+3MrOVxYxWeZp/ks8zTBAT6MSYkg\nY5iVUQlhmHy8XB1bRATo3D27S3PPRET6TI8FWlZWFqdOneLdd9+loKCA5cuX8+6773baJz8/n927\nd+Pjo8V6Ra5HkL8P09KjmZYeTVNzKzmnajiQZ+dAXiVbD5Wx9VAZJh8jtySFMzY1gjEpEQT56/8v\nEXGd/XmVnC5vbO+eRQS6Oo6IyIDVY4G2c+dO5s2bB0BycjJ1dXU0NjYSFPT1pQ2//vWv+elPf8rv\nfve7vksqMkCZfLwYmxLB2JQI2m53cKK0nv15dvblVbI3187eXDtGg4FhcSGMTbWSkRqBNdTf1bFF\nZBBpu6x7dre6ZyIifarHAq2yspK0tDTntsViwW63Owu0tWvXMmnSJGJiYvoupcggYTQaSIkNISU2\nhCVzUyirOsu+3PbO2rHTtRw7Xcs7m/KItQa1z1sbZiU+Mkg3GRGRPrU/105RRSNTRkUyRN0zEZE+\n9Y1vEuJwOJw/19bWsnbtWv7yl79QXl5+Xa8PCwvA2/vm59VYreabPkZ/8qS8yto3biSr1Wpm9Igo\nAKrrL5B15AyZR85wINfORztO8tGOk0SE+jMlLYrJ6VGkJ0fg7dU7C2MP9LF1FWUVT+PsnhngLt25\nUUSkz/VYoNlsNiorK53bFRUVWK1WAHbt2kV1dTXf/va3aWpq4vTp07zwwgssX768y+PV1Jy76dBW\nqxm7veGmj9NfPCmvsvaN3so6PiWc8SnhnL/YwpHCavbn2TmYX8X67YWs315IgK83o5PDyRhmJT3J\ngr/vjd2odTCObX8YjFlV5Hm+fcftFNvPMjUtkuhwdc9ERPpaj5/epk+fzmuvvcbSpUs5cuQINpvN\neXnjwoULWbhwIQDFxcX84he/6LY4E5He4e/rzYQRNiaMsNHS2kZeUS378io5kGdnV045u3LK8fYy\nMCIhjHGpVi2OLSI3pM3h4MPtl7pnmnsmItIfeizQxo0bR1paGkuXLsVgMLBixQrWrl2L2Wxm/vz5\n/ZFRRLrh7WVkZKKFkYkW/nVeKkUVjc55a9knqsk+Uc3fNhwnKTqYccMiGJtqZYgWxxaR67D3uJ0S\n+1mmpUcRZQlwdRwRkUHhuq5/+tnPftZpe8SIEVftExsbqzXQRFzMYDAQH2kmPtLMvTOHUll7nv35\nlezPbV8cu7CsY3HsMH8yUq1kDIsgeUiIFscWkau0ORys21aI0WDQ3DMRkX50YxNURMQjRIT6M39C\nHPMnXLY4dkdn7bOs03yWdRpzx+LY41KtjErU4tgi0m7PsQpKKs8yPT2KyDB1z0RE+osKNJFB4vLF\nsZtbWsk5WcP+vEoO5Fey7VAZ2zoWx05PCmfWuFiGRgZpcWyRQaqtrf3OjeqeiYj0PxVoIoOQj7cX\nY1IiGJMSQZvj68Wx9+dWsi/Xzr5cOwYDDIsNJSM1grHDrNi0OLbIoJF1rJyyqnPMuCUam7pnIiL9\nSgWayCBnNBhIiQkhJSaEJXPaF8fOLW1g24FicotqOV5Uyztf5hNrDXTOW0uINOsmIyIDVFubg4+2\nn8TLaGCxumciIv1OBZqIdBIdHsjoEVHMviWKusaLHMhvn7eWc7LGuTh2mNmXjNQIMlKtDI8P7bXF\nsUXc1Xvvvce6deuc29nZ2fz973/n2WefxWg0EhwczKpVq/D3/7rTvHbtWl599VXi4+MBmDZtGo89\n9li/Z/+mso62d89mjo5W51xExAVUoIlIl0KCfJk9NobZY2O40NRC9on2xbEPFVTx5b4SvtxXgv+l\nxbFTI7hlaPgNL44t4s6WLFnCkiVLAMjKyuLTTz/lueee4+mnn2b06NG89NJLrF27lm9/+9udXrdo\n0SKeeuopV0S+IW1tDtZ1dM/umpbo6jgiIoOSPkmJyHXxM129OPb+vPbuWmZOOZk55XgZDYxMCCNj\nmJWxKRGEmbU4tgw8r7/+OitXrsTf35+goCAALBYLtbW1Lk528zJzyjlTfY5ZY4YQoe6ZiIhLqEAT\nkW/s8sWxH+pYHLu9WLOTXVhNdmE1b204TlK0uX3eWmoEQyICNW9NPN6hQ4eIjo7GarU6Hzt37hwf\nfvghr7766lX7Z2Vl8eijj9LS0sJTTz3FqFGjuj1+WFgA3t43v9SF1Wr+xq9pbW3j412n8DIa+O7i\nNKz9tDD1jWR1JU/Kq6x9Q1n7jifl7cusKtBE5KZcvjj2PTOSqKw7z4GOztrx07UUljWw9p8nsIX5\nO+etpcRocWzxTGvWrOG+++5zbp87d47HHnuM73//+yQnJ3fad8yYMVgsFubMmcP+/ft56qmn+Oij\nj7o9fk3NuZvOaLWasdsbvvHrth8uo7TyLLPHDsHY2npDx/imbjSrq3hSXmXtG8radzwpb29k7a7A\nU4EmIr0qIsSfeRPimNexOPbhgir259k5fKKaDVlFbMgqci6OnZEaQVqiRYtji8fIzMzkmWeeAaCl\npYVly5axePFi7r///qv2TU5OdhZtGRkZVFdX09raipeX+/19b21r46MdHXdunJro6jgiIoOaCjQR\n6TNB/j5MTY9ianoUzS2tHD1V45y35lwc29tIWpKFjFQrY1LCMQeYXB1b5JrKy8sJDAzEZGr/O7p6\n9WomTZrkvHnIlVavXk10dDSLFy8mNzcXi8XilsUZwM7scipqzjMnI4bwED9XxxERGdRUoIlIv/Dx\n9mJ0cgSjkyP4zu0OCkvrnfPWLhVtBgOkdiyOnZEaoQVyxa3Y7XYsFotz++233yY2NpadO3cCMHny\nZJ544gkee+wx3njjDe666y5+/vOf884779DS0sLzzz/vqujdamlt46MdhXh7GVg8NcHVcUREBj0V\naCLS74wGA8kxISTHhPCtOcmUVZ11zlvLK6olt6iWd7/MJ+bS4tipESRGaXFsca309HTefPNN5/a2\nbduuud8bb7wBQFRUFG+99Va/ZLsZO4+cwV57gbnjYrAEq3smIuJqKtBExOWiwwOJDg/kjikJ1J1t\n4mB+Jftz7Rw5WcP6HSdZ37E49tiOztqI+DAtji3SC1pa2/ho+0m8vQzcOUXdMxERd6ACTUTcSkig\niVljhjBrzBAuI3VBRAAAEkRJREFUNLVwpLCafbmVHCqoZPO+EjbvK8Hf14tbhoaTkWplziR94y9y\no3Zkn6Gy7gK3jYtV90xExE2oQBMRt+Vn8mb8cBvjh9tobWsjt6iO/Xl2DuRVknW0gqyjFaxen8PQ\n6GDSkiykJ1lIjDbjZVR3TaQnLa1trN9xEm8vI4s090xExG2oQBMRj+BlNDIyIYyRCWE8dFv74tgH\n8io5VlTL8dM15JfU8eG2QgJ8vRmZGEZ6koW0JAsRIf6uji7ilrYfLmvvno2PJczs6+o4IiLSQQWa\niHicyxfHftRq5lRRNUdP1ZBdWE32iWr2Hrez97gdgEhLAOmJFtKGWhgRH4qfSf/siVzqnvl4G1mk\nuWciIm5Fn1RExOMF+Pk4L4V0OBxU1Jwnu7CaI4XVHD1dw6Z9xWzaV4yX0UBKTAjpQ9u7a/GRZoy6\nM6QMQtsOlVFVf5F5E9Q9ExG55KuvNjFnzm097vf888+zePEDDBkS0yc5VKCJyIBiMBiItAQQaQng\ntvGxtLS2UVBS5yzYcotqOV5Uy/tbThDk70NakoW0xPaCTR9UZTBoaW1j/U51z0RELldWVsoXX2y4\nrgLtl7/8JXZ7Q59lUYEmIgOat5eR4fFhDI8P44HZyTScayLnZA1HCqs5crKazJxyMnPKAYixBpKW\n2H6zkWFxoZh8vFycXqT3bT1URnX9RRZMjCM0SF9KiIgAvPzySxw9eoSZMyeyYMEdlJWV8sorv+fF\nF5/Fbq/g/PnzfP/7P2T69Jl85zvf4Ykn/jebN2/i7NlGTp8+RUlJMT/5yZNMnTr9prOoQBORQcUc\nYGLyqEgmj4rE4XBQWnmWI4XVZJ+sJvd0LRvtRWzcXdRe2MWFkJYUTlqShVhroBbKFo/X3NI+98zk\nbeSOyfGujiMick3/+DKf3ccqevWYE0fYePDWlC6ff+ih77B27T9ISkrm9OmT/P73b1JTU82kSVO4\n447FlJQU86tfPc306TM7va6iopyVK3/Lrl07+PDD91WgiYjcDIPBQIw1iBhrEAsmxdPc0kpucV17\nd62wmiMnazhysgY2t6/PltZxZ8i0RAvBgSZXxxf5xrYeKqWmob17FqLumYjINY0cmQaA2RzM0aNH\nWLduLQaDkfr6uqv2HT16LAA2m43GxsZe+fNVoImIdPDx9mqfj5ZogblQ23iRnJPVzoJtR/YZdmSf\nASA+Mqhj7bVwUmJC8PHW2mvi3ppbWvl456n27pnmnomIG3vw1pRuu119zcfHB4DPP/+M+vp6Xn/9\nTerr6/nBD75z1b5eXl9Ph3A4HL3y56tAExHpQmiQL9PSo5mWHk2bw0FxRWP75ZCF1eQV13K6vJFP\nd53G5GNkRHxY+/y1oRaiLAG6HFLczj8PllHTcJGFk+IJUQdYRKQTo9FIa2trp8dqa2uJjh6C0Whk\ny5YvaW5u7pcsKtBERK6D8bK11+6YksDFplaOF9V2FGxVHCpo/8UmsAT7dhRr4YxMCCPI38fV8WWQ\na++encTkY2Sh5p6JiFwlISGJ48ePER09hNDQUADmzLmVp5/+3+TkZHPnnXdjs9n4y19W93kWFWgi\nIjfA1+TF6ORwRieHA6lU119wdtdyTlaz9VAZWw+VYQASo4NJS7IwIyMWS4A33l66HFL611cHSqlt\nbOKOyfGaPykicg1hYWGsXftxp8eio4fw17++49xesOAOAKxWM3Z7A0OHfn0Z5tChKfzud3/qlSwq\n0EREeoEl2I+ZY4Ywc8wQ2tocnCpvIPtEFUcKqykoraewrJ71O07iZ/JiZEIY6R03HLGFBbg6ugxw\nTc2tfLLrFL4+XuqeiYh4ABVoIiK9zGg0kBQdTFJ0MHdNT+L8xRaOna6hoKyBPUfL2Z9Xyf68SgCs\noX6kd9zKf0R8GAF++mdZeteWA6XUNTaxaEoC5gB1z0RE3J0+CYiI9DF/X28yUq0smDYUu72Bitrz\nzjtDHj1Vzeb9JWzeX4LRYCA5Jth5O/+kqGCMRt1sRG6cs3tm8uL2SXGujiMiItdBBZqISD+zhfpj\ny4hhbkYMrW1tnCitdxZs+SV15BXX8cHWQgL9vBmZaGm/HDLRQniIn6uji4f5an8JdWebuHOqumci\nIp5CBZqIiAt5GY2kxoaSGhvKvTOHcvZCM0dP1pBdWM2Rwir2HKtgz7EKAKLDA9rXaeu4HNLX5NXD\n0WUwu9jcyieZp/EzeXH7JM09ExHxFCrQRETcSKCfDxNG2JgwwobD4eBM9Tlnd+3Y6Vq+2FvMF3uL\n8TIaSI0NcS6WHRcZhFFrr8llNu8rof5sE4unJWipBxERD6ICTUTETRkMBqLDA4kOD2TehDiaW9oo\nKKnjyMlqsk+0F2zHTtfy/pYTmAN8nN21tCQLoUG+ro4vLnSxqZVPM0/h7+vFgonqnomI9JZvfesu\nPvnk4553vAkq0EREPISPt5ERCWGMSAjjgdnJ1J9rIudkNUdOVJN9sppdOeXsyikHINYa6OyupcaG\nYPLR5ZCDyeb9JTSca+auaYnqnomIeBgVaCIiHio4wMSUUVFMGRWFw+GgpPKsc7Hs3KJaiu1FbMgq\nwsfbyLC4UOfaazERgRh0OeSAdeFiS0f3zJsFunOjiMh1+f73v80LL6wiKiqKM2fK+MUvnsRqtXH+\n/HkuXLjAT3/6c0aNSu+XLCrQREQGAIPBQKw1iFhrELdPiqepuZW84jqyC6ucc9iOFFYDEBpkcl4K\nOSrRQrDu7jegfLy9kIZzzdw9PZFAP3XPRMTzrM1fz/6Kw716zAzbLdyfsrjL52fNmsv27f/kgQce\nZOvWLcyaNZfk5FRmzZrD3r27efvtv/L88/+nVzN1RQWaiMgAZPLxchZhADUNF9svhyys5sjJarYf\nPsP2w2cASIg0kz60/Vb+KbEheHsZXRldbsKFphbWfpXf3j2bqO6ZiMj1mjVrLr/73Ss88MCDbNu2\nhSee+CnvvPMWf//7WzQ3N+Pn139L3ahAExEZBMLMvky/JZrpt0TT5nBQVN7o7K7lFddxqryBj3ee\nwtfHixHxocwaH8eYpDDdGbLDe++9x7p165zb2dnZ/P3vf+c///M/ARg+fDj/9V//1ek1zc3NPP30\n05SWluLl5cWLL75IXFzfFk2b9hZTf7aJe2YkEaDumYh4qPtTFnfb7eoLQ4cmU1Vlp7z8DA0NDWzd\n+hURETZ+9av/j2PHcvjd717ptywq0EREBhmjwUBClJmEKDN3Tk3kQlMLx0/XOrtrBwuqOFhQxa9/\nNAVbWICr47qFJUuWsGTJEgCysrL49NNPef7551m+fDmjR4/mySefZMuWLcyePdv5mvXr1xMcHMyq\nVavYtm0bq1at4pVX+vYEf6SwGnOAD/MnqHsmIvJNTZ06gz/96ffMnDmb2toakpNTAdiyZTMtLS39\nlkMFmojIIOdn8mZMSgRjUiIAqKq7AD5ehAeoA3Mtr7/+Oi+++CIPP/wwo0ePBmDu3Lns3LmzU4G2\nc+dO7r33XgCmTZvG8uXL+zzbDxaPIiQ0AK+2tj7/s0REBprZs+fy4x9/n//+779z4cJ5nntuBZs3\nf8EDDzzIF19s5OOP1/V8kF6gAk1ERDoJD/HDajVjtze4OorbOXToENHR0Xh5eREcHOx8PDw8HLvd\n3mnfyspKLJb2OYBGoxGDwUBTUxMmU9c3ZQkLC8Db+8aXRLBazTf8WlfxtMyelFdZ+4ay9h1X57Va\np5CTk+Pc3rhxg/Pn++5rv+TykUe+DUBgYN/lUIEmIiJyndasWcN999131eMOh6PH117PPjU1524o\n1+U8qbj2pKzgWXmVtW8oa9/xpLy9kbW7YlS36hIREblOmZmZZGRkYLFYqK2tdT5eXl6OzWbrtK/N\nZnN21Zqbm3E4HN12z0REREAFmoiIyHUpLy8nMDAQk8mEj48PQ4cOZc+ePQBs3LiRmTNndtp/+vTp\nfPbZZwBs3ryZyZMn93tmERHxPCrQREREroPdbnfOKQNYvnw5L7/8MkuXLiU+Pp5p06YB8NhjjwGw\naNEi2traeOihh3j77bd58sknXZJbREQ8i+agiYiIXIf09HTefPNN53ZKSgr/8z//c9V+b7zxBoBz\n7TMREZFvQh00ERERERERN6ECTURERERExE2oQBMREREREXETKtBERERERETchAo0ERERERERN2Fw\nOBwOV4cQERERERERddBERERERETchgo0ERERERERN6ECTURERERExE2oQBMREREREXETKtBERERE\nRETchAo0ERERERERN+Ht6gA9eeGFFzh48CAGg4Hly5czevRo53M7duzg5ZdfxsvLi1mzZvH444+7\nMGn3WW+99VaioqLw8vICYOXKlURGRroqKgC5ubksW7aMRx55hIcffrjTc+42tt1ldbex/c1vfsPe\nvXtpaWnhRz/6EQsWLHA+527j2l1WdxrX8+fP8/TTT1NVVcXFixdZtmwZc+fOdT7vTuPaU1Z3GtfL\nXbhwgcWLF7Ns2TLuv/9+5+PuNLbSmSedH8GzzpE6P/YdnSN7n86Rfcsl50eHG8vMzHT88Ic/dDgc\nDkd+fr7jwQcf7PT8HXfc4SgtLXW0trY6HnroIUdeXp4rYjocjp6zzp0719HY2OiKaNd09uxZx8MP\nP+x45plnHG+99dZVz7vT2PaU1Z3GdufOnY4f/OAHDofD4aiurnbMnj270/PuNK49ZXWncf34448d\nf/rTnxwOh8NRXFzsWLBgQafn3Wlce8rqTuN6uZdfftlx//33O95///1Oj7vT2MrXPOn86HB41jlS\n58e+o3Nk39A5sm+54vzo1pc47ty5k3nz5gGQnJxMXV0djY2NABQVFRESEkJ0dDRGo5HZs2ezc+dO\nt8zqjkwmE6tXr8Zms131nLuNbXdZ3c3EiRN59dVXAQgODub8+fO0trYC7jeu3WV1N4sWLeLf/u3f\nACgrK+v0bZq7jWt3Wd1VQUEB+fn5zJkzp9Pj7ja28jVPOj+CZ50jdX7sOzpH9g2dI/uOq86Pbn2J\nY2VlJWlpac5ti8WC3W4nKCgIu92OxWLp9FxRUZErYgLdZ71kxYoVlJSUMH78eJ588kkMBoMrogLg\n7e2Nt/e1//O729h2l/USdxlbLy8vAgICAFizZg2zZs1ytundbVy7y3qJu4zrJUuXLuXMmTP84Q9/\ncD7mbuN6ybWyXuJu4/rSSy/xq1/9ig8++KDT4+46tuJZ50fwrHOkzo99R+fIvqVzZO9z1fnRrQu0\nKzkcDldHuG5XZv3JT37CzJkzCQkJ4fHHH2fDhg0sXLjQRekGFncc2y+++II1a9bw5z//2aU5rkdX\nWd1xXN955x2OHj3Kz3/+c9atW+fyk2F3usrqbuP6wQcfMHbsWOLi4lyWQW6eJ50fQefI/uKu46pz\nZN/QObJ3ufL86NaXONpsNiorK53bFRUVWK3Waz5XXl7u0hZ/d1kB7r33XsLDw/H29mbWrFnk5ua6\nIuZ1cbex7Ym7je3WrVv5wx/+wOrVqzGbzc7H3XFcu8oK7jWu2dnZlJWVATBy5EhaW1uprq4G3G9c\nu8sK7jWuAF999RWbNm3iwQcf5L333uP3v/89O3bsANxvbOVrnnR+hIFzjnTHse2OO46rzpG9T+fI\nvuHK86NbF2jTp09nw4YNABw5cgSbzea8HCI2NpbGxkaKi4tpaWlh8+bNTJ8+3S2zNjQ08Oijj9LU\n1ATA7t27SU1NdVnWnrjb2HbH3ca2oaGB3/zmN/zxj38kNDS003PuNq7dZXW3cd2zZ4/z28vKykrO\nnTtHWFgY4H7j2l1WdxtXgFdeeYX333+ff/zjHyxZsoRly5Yxbdo0wP3GVr7mSedHGDjnSHcc2664\n47jqHNk3dI7sG648Pxocbn5dxMqVK9mzZw8Gg4EVK1aQk5OD2Wxm/vz57N69m5UrVwKwYMECHn30\nUbfN+te//pUPPvgAX19fRo0axa9+9SuXtp6zs7N56aWXKCkpwdvbm8jISG699VZiY2Pdbmx7yupO\nY/vuu+/y2muvkZSU5Hxs8uTJDB8+3O3Gtaes7jSuFy5c4Je//CVlZWVcuHCBJ554gtraWrf8t6Cn\nrO40rld67bXXiImJAXDLsZXOPOn8CJ5zjtT5se/oHNk3dI7se/19fnT7Ak1ERERERGSwcOtLHEVE\nRERERAYTFWgiIiIiIiJuQgWaiIiIiIiIm1CBJiIiIiIi4iZUoImIiIiIiLgJFWgiIiIiIiJuQgWa\niIiIiIiIm1CBJiIiIiIi4ib+H73+Z5wGDsqVAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<matplotlib.figure.Figure at 0x7fb0175d8a58>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "metadata": {
        "id": "4EmFhiX-FMaV",
        "colab_type": "code",
        "outputId": "42d7dcac-f4cb-4bf2-e0e9-c3ee2007df1d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "cell_type": "code",
      "source": [
        "# Test performance\n",
        "trainer.run_test_loop()\n",
        "print(\"Test loss: {0:.2f}\".format(trainer.train_state['test_loss']))\n",
        "print(\"Test Accuracy: {0:.1f}%\".format(trainer.train_state['test_acc']))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Test loss: 0.48\n",
            "Test Accuracy: 83.2%\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "zVU1zakYFMVF",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Save all results\n",
        "trainer.save_train_state()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "qLoKfjSpFw7t",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Inference"
      ]
    },
    {
      "metadata": {
        "id": "tDu1oFRsWPlr",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "class Inference(object):\n",
        "    def __init__(self, model, vectorizer, device=\"cpu\"):\n",
        "        self.model = model.to(device)\n",
        "        self.vectorizer = vectorizer\n",
        "        self.device = device\n",
        "  \n",
        "    def predict_category(self, dataset):\n",
        "        # Batch generator\n",
        "        batch_generator = dataset.generate_batches(\n",
        "            batch_size=len(dataset), shuffle=False, device=self.device)\n",
        "        self.model.eval()\n",
        "        \n",
        "        # Predict\n",
        "        for batch_index, batch_dict in enumerate(batch_generator):\n",
        "            # compute the output\n",
        "            y_pred =  self.model(batch_dict['title'], batch_dict[\"title_length\"],\n",
        "                                 apply_softmax=True)\n",
        "\n",
        "            # Top k nationalities\n",
        "            y_prob, indices = torch.topk(y_pred, k=len(self.vectorizer.category_vocab))\n",
        "            probabilities = y_prob.detach().to('cpu').numpy()[0]\n",
        "            indices = indices.detach().to('cpu').numpy()[0]\n",
        "\n",
        "            results = []\n",
        "            for probability, index in zip(probabilities, indices):\n",
        "                category = self.vectorizer.category_vocab.lookup_index(index)\n",
        "                results.append({'category': category, 'probability': probability})\n",
        "\n",
        "        return results"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "KGsxAknUWjgT",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Load vectorizer\n",
        "with open(args.vectorizer_file) as fp:\n",
        "    vectorizer = NewsVectorizer.from_serializable(json.load(fp))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "dvkAgy6mWjdC",
        "colab_type": "code",
        "outputId": "1c63df18-817b-4435-c3c4-4a4116529822",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 136
        }
      },
      "cell_type": "code",
      "source": [
        "# Load the model\n",
        "model = NewsModel(embedding_dim=args.embedding_dim, \n",
        "                  num_embeddings=len(vectorizer.title_vocab), \n",
        "                  rnn_hidden_dim=args.rnn_hidden_dim,\n",
        "                  hidden_dim=args.hidden_dim,\n",
        "                  output_dim=len(vectorizer.category_vocab),\n",
        "                  num_layers=args.num_layers,\n",
        "                  bidirectional=args.bidirectional,\n",
        "                  dropout_p=args.dropout_p, \n",
        "                  pretrained_embeddings=None, \n",
        "                  padding_idx=vectorizer.title_vocab.mask_index)\n",
        "model.load_state_dict(torch.load(args.model_state_file))\n",
        "print (model.named_modules)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<bound method Module.named_modules of NewsModel(\n",
            "  (embeddings): Embedding(3406, 100, padding_idx=0)\n",
            "  (gru): GRU(100, 128, batch_first=True)\n",
            "  (dropout): Dropout(p=0.1)\n",
            "  (fc1): Linear(in_features=128, out_features=100, bias=True)\n",
            "  (fc2): Linear(in_features=100, out_features=4, bias=True)\n",
            ")>\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "oUBYxSEfWjZy",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Initialize\n",
        "inference = Inference(model=model, vectorizer=vectorizer, device=args.device)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "KXHyLE_PWjXZ",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "class InferenceDataset(Dataset):\n",
        "    def __init__(self, df, vectorizer):\n",
        "        self.df = df\n",
        "        self.vectorizer = vectorizer\n",
        "        self.target_size = len(self.df)\n",
        "\n",
        "    def __str__(self):\n",
        "        return \"<Dataset(size={1})>\".format(self.target_size)\n",
        "\n",
        "    def __len__(self):\n",
        "        return self.target_size\n",
        "\n",
        "    def __getitem__(self, index):\n",
        "        row = self.df.iloc[index]\n",
        "        title_vector, title_length = self.vectorizer.vectorize(row.title)\n",
        "        return {'title': title_vector, 'title_length': title_length}\n",
        "\n",
        "    def get_num_batches(self, batch_size):\n",
        "        return len(self) // batch_size\n",
        "\n",
        "    def generate_batches(self, batch_size, shuffle=True, drop_last=False, device=\"cpu\"):\n",
        "        dataloader = DataLoader(dataset=self, batch_size=batch_size, \n",
        "                                shuffle=shuffle, drop_last=drop_last)\n",
        "        for data_dict in dataloader:\n",
        "            out_data_dict = {}\n",
        "            for name, tensor in data_dict.items():\n",
        "                out_data_dict[name] = data_dict[name].to(device)\n",
        "            yield out_data_dict"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "3FXcmtXWWjUY",
        "colab_type": "code",
        "outputId": "4a7e757d-b1e9-42c8-8f84-ac7029c2146f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 102
        }
      },
      "cell_type": "code",
      "source": [
        "# Inference\n",
        "title = input(\"Enter a title to classify: \")\n",
        "infer_df = pd.DataFrame([title], columns=['title'])\n",
        "infer_df.title = infer_df.title.apply(preprocess_text)\n",
        "infer_dataset = InferenceDataset(infer_df, vectorizer)\n",
        "results = inference.predict_category(dataset=infer_dataset)\n",
        "results"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Enter a title to classify: President Obama signed the petition during the White House dinner.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[{'category': 'World', 'probability': 0.40057328},\n",
              " {'category': 'Business', 'probability': 0.30899048},\n",
              " {'category': 'Sci/Tech', 'probability': 0.27638948},\n",
              " {'category': 'Sports', 'probability': 0.01404677}]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 110
        }
      ]
    },
    {
      "metadata": {
        "id": "1YHneO3SStOp",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# TODO"
      ]
    },
    {
      "metadata": {
        "id": "gGHaKTe1SuEk",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "- interpretability with task to see which words were most influential"
      ]
    }
  ]
}